ORMS Today
February 2000

Criterium Decision Plus 3.0

Versatile multi-criteria tool excels in its ability to support decision-making

By Walt Haerer

InfoHarvest Inc.'s (www.infoharvest.com) latest version of Criterium Decision Plus combines analytical power, ease of use and an extensive graphics interface that together make CDP transparent to first-time audiences yet meet the demands of sophisticated analysts. Some examples of known CDP use will demonstrate this versatility. These projects lasted from a few hours to several months, involved from one decision-maker to 40 participating organizations, were conducted in the United States and overseas, and evaluated costs of as little as $100,000 to more than $1 billion. The projects:
  • Selecting a jet-fuel pipeline route through sensitive marine habitats for a new, major international airport.
  • Evaluating buy/lease/sell/build options for the corporate headquarters of a major oil company.
  • Restructuring and consolidating first-line organizations for a government agency.
  • Selecting a Fortune 500 company's business system (hardware/software vendors).
  • Consolidating database facilities of an aerospace company.
  • Prioritizing regional wastewater re-use projects.
  • Selecting remedial strategies and technologies for many hazardous waste sites.
  • Prioritizing environmental upgrades for a large smelting operation and for a pulp and paper plant.
  • Working with multiple stakeholders to select a habitat restoration strategy requiring the potential relocation of hundreds of families.
  • Evaluating municipal waste management options for a large metropolitan area.
  • Prioritizing major capital improvement projects with input from multiple stakeholder groups.
  • Helping a federal agency evaluate policy alternatives.
  • Evaluating bidders for a risky public works project.
  • Selecting a new home.
  • Evaluating career alternatives.
CDP's primary strengths are in providing:
  • A structured decision-making framework that is totally transparent; the basic algorithms can be displayed on a spreadsheet.
  • Immediate, graphic feedback from what-if analysis.
  • Easy evaluation of sensitivities of the rankings of alternatives to weights/trade-offs to help groups focus on key issues.
  • Ability to incorporate uncertainties in performance scores (and thereby defuse potentially extensive arguments among experts).
  • Graphic portrayal of sensitivities of rankings to uncertainties in performance scores and support of value of information analyses.
  • Consensus building.
  • Minimizing the time and cost of reaching closure on major decisions.
  • Full verbal and graphic documentation of all input to the decision process.
CDP has been used live on the Internet and has supported decision-making via video conferencing. The reader is encouraged to view some of these benefits of CDP directly by visiting InfoHarvest's website (www.infoharvest.com) for free downloads, examples, tips and a "best practices center" that currently provides a systems engineering example.

Technical Background

CDP is really two models. Users can select either a simple multi-attribute rating technique [Edwards, 1994; von Winterfeldt and Edwards, 1986] or the Analytic Hierarchy Process [Saaty, 1992]. This review will focus on the simple multi-attribute rating technique; readers can refer to the excellent CDP manual for details concerning application of the Analytic Hierarchy Process.

The features of CDP are best illustrated with a simple example that has been used hundreds of times by the author both for introducing audiences to multi-criteria decision-making and as part of training. The example was chosen to be simple and non-controversial so that audiences can focus on the process and functionality of CDP and not get intellectually or emotionally involved in the decision itself.

Model Structuring

For purposes of this example, assume that the goal of the decision opportunity or problem is clear: select replacement tires for our car. CDP's brainstorm screen provides a means to capture and connect candidate objectives or criteria such as minimizing cost, maximizing safety, maximizing wear, etc. to the overall goal (see Figure 1A/B). We've selected only three objectives from among many possible ones. We've also eliminated required tire characteristics such as size. Cost and wear will be measured directly and safety will have traction the distance required to stop a car as a surrogate measure. These criteria should be essential, measurable, nonredundant, etc. as outlined by Keeney [1992].

Figure 1A

Figure 1A: "Brainstorming" objectives.

Although several layers of criteria, sub-criteria, sub-sub-criteria, etc. are possible in CDP, only one layer will be used here (see the InfoHarvest website procurement model for a 2-layer example).

Figure 1B

Figure 1B: A tale of three choices.

Many tire choices are available to us, but we've selected only three and listed them as shown in Figure 1A/B. CDP can handle up to 200 alternatives or a total of 500 blocks. If we had a spreadsheet table of decision criteria, alternatives and scores, we could import that table directly to the brainstorm screen. Once we've agreed on a set of criteria and alternatives, we generate a hierarchy automatically as depicted in Figure 1A/B. CDP provides several options for presenting graphic information: font size, spacing between criteria/alternatives, automatic resizing, automatic connection of alternatives to criteria, decimal places, etc. Definitions of criteria and alternatives can be documented and retrieved by passing the cursor over marked criteria and alternatives.

Performance Measurement

Our next task is to create scales that reflect the performance of the alternatives with respect to each criterion. Quantitative and qualitative scales can be created in CDP and several default scales are also available. For this tire replacement example, we've decided to create quantitative scales for cost ($50 to $100/tire), wear (50 to 100K miles), and traction (100 to 150 feet to stop a car at 60 mph). The scales will automatically create default linear value functions that reflect the best and worst conditions measured by the scales; value functions will be positive for wear and negative for cost and traction. CDP enables users to select non-linear value functions if necessary or desirable. Performance scores can be entered into a simple data table or in a rating window (Figure 2). The rating window provides choices of numerical, graphic and verbal representations of performance scores.

Figure 2

Figure 2: Rating performance.


If there is lack of knowledge or known variability in performance scores, or if there is disagreement about scores, CDP's uncertainty feature can be used to reflect and quantify these uncertainties. The author has found this feature to not only help clients focus on critical uncertainties, but also to help defuse disagreements about "best estimates" by capturing disagreements with uncertainty distributions, and then determining if these disagreements matter. CDP provides several choices including uniform, triangular, normal, lognormal and custom distributions and these are graphically portrayed. For the tire example, triangular distributions were used to capture uncertainties in performance scores (Figure 3).

Figure 3

Figure 3: Triangular distributions capture uncertainties.

Weights & Trade-offs

One of the most useful and powerful functions of CDP is in the next step, assigning weights and making trade-offs among the criteria. The analyst has options for using any one or more of the traditional approaches to assigning weights, or going directly to trade-offs that will automatically create appropriate weights. In any case, the trade-off functionality can be used to test the reasonableness and validity of weights assigned directly (Figure 4). The author has found time and again that decision-makers, and especially groups, need feedback about trade-offs produced by combinations of weights and scales no matter what process was used to elicit weights; often the original weights produced wildly unacceptable trade-offs. In the tire example, we arbitrarily assigned weights that result in trade-offs of $1/foot traction/3,000 miles wear (see Figure 4A/B).

Figure 4A

Figure 4A: Weighty trade-offs.

Figure 4B

Figure 4B: Weighty trade-offs.

Evaluation of Preliminary Results & Sensitivity Analyses

The first iteration of the decision process results in rankings of alternatives that can be examined in several ways with CDP. Rankings, contributions of criteria and sensitivity analyses are all graphically presented as in Figures 5 and 6. CDP provides several graphic aids to depict contributions of criteria in addition to the stacked histogram shown here: pie charts, scatter diagrams and radar charts. In the tire example, Tires B and C clearly score higher than Tire A and the contributions of criteria to these rankings are shown in Figure 5.

Figure 5

Figure 5: Ranking alternatives.

Sensitivity of rankings to weights/trade-offs can be depicted simultaneously with the rankings of alternatives and manipulated live to show how increases/decreases in weights influence rankings (Figure 6). This feature of CDP has contributed significantly to gaining group consensus and to explaining the rationale for decisions to senior management. In the tire example, rankings of the top performing tires are somewhat sensitive to changes in all three criteria, whereas the lowest ranking alternative would require all tradeoffs to be subordinated dramatically to the "wear" criterion.

Figure 6

Figure 6: Sensitivity to weighting changes.

Uncertainty Analyses

Uncertainties in the weighted decision scores can be shown in two ways: 1. figures of cumulative, reverse cumulative or density functions, or 2. as uncertainty bars superimposed on rankings based on best estimates (Figure 7A/B). The bars depicted in Figure 8 cover the 5th through 95th percentiles and depict the mean value as a single bar. The percentile figures to the right indicate what proportion of the time an alternative is the best decision. Note that if uncertainty were ignored and tires were selected solely on the basis of "best estimates," then the selection of Tire B would be "wrong" about three quarters of the time.

Figure 7A

Figure 7A: Uncertainties exposed.

Figure 7B

Figure 7B: Uncertainties superimposed.

At this stage, most clients would want to know how much of a reduction in which uncertainties would change the rankings of alternatives. CDP provides a "Contribution to Uncertainty" window that shows how much uncertainties in each criterion contribute to the overall uncertainty in the scores of the alternatives (Figure 8). In the case of the tire example, uncertainties in traction and in cost contribute most to the overall uncertainty in decision scores. What-if analyses are then conducted to illustrate if, and how much, the rankings would change if uncertainties could be reduced by some specific amount. These analyses provide a measure of the value of collecting additional information; in the tire example, reducing cost uncertainty has very little effect on rankings under uncertainty. In a real decision situation, the cost, time and resources required to reduce uncertainty to a point that is technically feasible and that would result in a change in rankings could then be compared to the benefits of collecting that information.

Figure 8

Figure 8: Contribution to uncertainty analysis.


All steps of any decision process supported by CDP can be documented directly and connected to criteria, alternatives and the overall goal. Furthermore, because of the information contained in the many graphic displays provided by CDP, reports can capture all critical steps of a decision process and thereby enhance the value of the report to key decision makers and, if desired, to stakeholders and the public.

Product Information

CDP is supported by a comprehensive User's Guide and can be installed in less than five minutes. This 32-bit version requires Microsoft Windows 95, 98 or NT 4 (SP3 or higher) and Windows 2000. The InfoHarvest website provides guidance about dealing with potential issues when installing on systems running Microsoft NT, especially for users who might not have administrator privileges on their systems. InfoHarvest recommends at least 16MB RAM, 14MB disk space, a CD-ROM drive, VGA or SVGA (the latter recommended), and a mouse.

The program can be learned quickly with help of the tutorial and by downloading examples from InfoHarvest's website. The author has trained hundreds of individuals who were unfamiliar with decision analysis to use 90 percent of the CDP functionality in one to two days. Decision analysts can learn most of CDP in one to two hours.


Criterium Decision Plus can be used to advantage in the classroom, at home, and by OR/MS practitioners who support government agencies and industry, and especially by those who facilitate consensus building in any context where more than one objective is at issue. CDP has been used live on the Internet and has supported decision-making via video conferencing.

Product Summary
Criterium Decision Plus is produced by:
InfoHarvest, Inc.
PO Box 25155, Seattle, WA 98125-2055
E-mail: Info@InfoHarvest.com
Phone: 1-800-869-7150
Fax: 1-206-622-8430

CDP: $595
Upgrade to 3.0.2: $245
Free Demos Available at http://www.infoharvest.com/infoharv/ CDPFreeDownloads.htm

Site Licenses Available
Academic Pricing Available


  1. Edwards, W. and F.H. Barron, 1994, "SMARTS and SMARTER: Improved Simple Methods for Multiattribute Utility Measurement," Organizational Behavior And Human Decision Processes, Vol. 60, pp. 306-325.
  2. Keeney, R.L., 1992, "Value-Focused Thinking: A Path to Creative Decisionmaking," Harvard University Press, Cambridge, Mass.
  3. Saaty, T.L., 1992, "Multicriteria Decision Making - The Analytical Hierarchy Process," RWS Publications, Pittsburg, Pa.
  4. von Winterfeldt, D. and W. Edwards, 1986, "Decision Analysis and Behavioral Research," Cambridge University Press, New York, N.Y.

Walt Haerer provides decision support, risk assessment and risk management services to industrial and government clients in the United States and abroad. He has used Criterium Decision Plus since 1993.

  • Table of Contents

  • OR/MS Today Home Page

    OR/MS Today copyright 2000 by the Institute for Operations Research and the Management Sciences. All rights reserved.