August 1996 € Volume 23 € Number 4
Expert Choice
Pro Version 9.0 for Windows confirms product as an outstanding choice for addressing complex, multicriteria problems
By Abel A. Fernandez
Expert Choice Pro Version 9.0 for Windows is a multiattribute decision support software tool based on the analytic hierarchy process (AHP) methodology developed by Thomas L. Saaty. This product is the flagship of the decision support products available through Expert Choice Inc. The Expert Choice product has been available for more than 12 years. This latest release provides several new features and is designed as a Windows application. Version 8.0 of Expert Choice, a DOS based product, was reviewed in the June 1993 issue of OR/MS Today [Bahouth, 1993].
Expert Choice Pro helps a decision maker examine and resolve problems involving multiple evaluation criteria. The software uses the AHP methodology to model a decision problem and evaluate the relative desirability of alternatives. Dr. Saaty (developer of AHP) is co-founder of Expert Choice Inc. and still active within the firm, so there is full confidence in the technical soundness of Expert Choice Pro with respect to the AHP methodology.
The active research on many aspects of this technique aside, AHP has become increasingly popular among practitioners for its comprehensible and insightful analysis of complex problems. The academic and practitioner literature cites hundreds (if not thousands) of research and application papers involving AHP. Reported applications include problems in public policy, marketing, procurement, health care, corporate planning, transportation planning and many other areas. A recent Interfaces article by Thomas Saaty [Saaty, 1994] provides an excellent overview of the AHP and its applicability to complex multicriteria problems. Due to its relatively widespread familiarity within the OR/MS community, the theory, history and present state of the AHP will not be discussed in this review.
Product Overview
Expert Choice Pro is a Windows application requiring a minimum of 8 MB of RAM and 10 MB of disk space. As part of this review, the program was run under both Windows and Windows 95 without encountering problems under either environment. Version 9.0 migrates the previous DOS release to Windows and, according to the product literature, incorporates 15 new capabilities and features, including enhanced interface capabilities and the new Structuring module.
The Expert Choice Pro software is grouped into three principal software modules: the Online Tutorial, the Structuring module, and the Evaluation and Choice module. The Online Tutorial is a well-designed software version of the hard copy tutorial manual provided with the product. The Structuring module is a feature designed to assist users in creating an AHP model of the decision problem. It is a self-help tool for problem understanding and hierarchical modeling. Although models can be directly created within the Evaluation and Choice module, the Structuring module provides a mechanism for facilitating problem understanding.
The Evaluation and Choice module is the principal component of Expert Choice Pro. This module is used for creating a model (alternatively a model can be imported from a previous session with the Structuring module or imported from a library of standard model formats), eliciting expert comparison assessments, solving a model, performing sensitivity analysis, and generating reports.
Expert Choice Pro decision models follow the standard AHP format, a functional hierarchy with the broad overall objective (or goal) at the highest level (known as level zero). Lower levels correspond to the criteria and respective subcriteria used to choose among alternatives.
At the lowest level of the hierarchy are the alternatives to be evaluated. Figure 1 illustrates a small model for selecting a site location for an ice cream store. The goal is to select a site from three possible alternatives: a shopping center, a downtown location and a mall location. The four criteria under the goal represent the principal considerations for making the selection. Note that this particular hierarchy is not symmetrical since the Cost criterion is further divided into the subcriteria of initial and monthly costs. The three alternatives are at the lowest level of the hierarchy.
Figure 1
Very large AHP models can be created using Expert Choice Pro, since Version 9.0 allows up to nine children nodes for each parent node. Even larger models can be accommodated by a technique of clustering and linking between nodes. Very large models, however, impose significant effort in eliciting pairwise comparison assessments. For these situations, Expert Choice Pro provides a Ratings capability in which alternatives are not compared against each other but are compared against standards or norms.
The Expert Choice Pro software assists the user in all phases of the problem solving process, from model formulation to final report output. The following sections describe the three principal modules of the software.
The Online Tutorial
Supplementing the hard copy documentation provided with the software (a User's Guide, quick reference card and Tutorial Manual), the Online Tutorial provides an interactive overview of the product's principal features. A first-time user can quickly become familiar with the software through the Online Tutorial (which takes less than one hour to complete). There are three particularly good features of the Online Tutorial: 1. the user learns by example; 2. procedures are demonstrated using Lotus Screen Cam; and 3. the user can easily switch between the tutorial and the actual program.
Learning by example concerns the method of instruction adopted by the Online Tutorial: product features are illustrated by focusing on a single, simple application. The user follows the entire problem-solving process for this example.
Lessons consist of hypertext descriptions complemented by Lotus Screen Cam animated demonstrations. This multimedia approach is well conceived and implemented. Furthermore, at the completion of each lesson the Online Tutorial encourages the user to practice the concepts on the actual program. This encouragement is an effective reinforcement of the lessons learned throughout each of the tutorial steps.
The Online Tutorial is accompanied by a tutorial manual. Although the manual is comprehensive and useful, it contains a number of typographical and other editorial errors which detract from the professional impact of the product.
The Structuring Module
Expert Choice Pro provides two methods of developing a decision model: direct construction using the Evaluation and Choice module, and assisted construction using the Structuring module. (The Expert Choice Pro documentation states that one may also "use an existing model from the model library," however without detailed descriptions these models were not very useful starting points for specific applications.)
The Structuring module provides an interface mechanism for deriving criteria and subcriteria. It is a framework for collecting ideas and transforming these into an AHP model, a facilitating mechanism for constructing models. In a bottom-up approach, criteria are developed by listing pros and cons of individual alternatives, grouping these into clusters and converting each cluster into a generic objective (criteria). The criteria are then organized into the hierarchy of an AHP model. Models can also be built using a top-down approach, in which objectives (criteria) are defined and then organized into an AHP hierarchy. The Structuring module provides user interfaces for creating a hierarchy under either a top-down or bottom-up approach, and hooks for linking the model with the Evaluation and Choice module.
The technique of identifying pros and cons of alternatives and transforming these into criteria is very useful for conceptualizing an AHP model. However, although practical for small problems, the Structuring module's interface quickly became cumbersome for moderately sized problems. This was principally due to the physical limitations associated with a 13-inch monitor.
The Evaluation and Choice Module
The Evaluation and Choice module is the principal component of Expert Choice Pro. This module provides the facilities for model creation, pairwise comparisons, solution synthesis, sensitivity analysis and report generation. Version 9.0 for Windows includes many interface enhancements over the previous release, including toolbars, more aesthetically pleasing window format, folder and tab style presentation and other user-oriented features. In general, the design is effective and simple to learn and use. Special symbols and icons are kept to a minimum, and where used they are logical and easy to interpret.
In lieu of using the Structuring module, models can be directly input using the Evaluation and Choice module. The facilities for entering and modifying model data are generally very good. Models can be easily created using straightforward commands and icons. Restricting names to eight characters, however, became a minor annoyance.
The next step after constructing the model is to perform comparison assessments leading to the AHP solution synthesis. Expert Choice Pro provides excellent facilities for performing these comparisons using either of two modes. The pairwise comparison mode is recommended for problems with small numbers of alternatives. In this mode, elements are pairwise compared against each other with respect to a higher level element.
The required comparisons rapidly become impractical in problems with large numbers of alternatives, therefore Expert Choice Pro also provides a ratings comparison mode. In the ratings mode, alternatives are compared not against specific criteria but against norms (standards). This approach reduces the number of required comparisons, and makes it possible to apply the AHP technique to problems with large numbers (potentially in the hundreds) of alternatives.
Expert Choice Pro provides excellent facilities for eliciting expert assessments. In order to accommodate different types of models (having different classes of criteria), pairwise comparisons can be made using combinations of different comparison types and modes. The three different comparison types are importance , preference, and likelihood. For any of these types, assessments may be made using either a verbal mode (comparisons are based on English language descriptors), a graphical mode (comparisons are based on visual expressions of preference), or a numerical mode (comparisons are based on numerical measures).
The numerical mode is particularly useful for eliciting expert opinion from third parties since hard-copy questionnaires can be produced for distribution (a standard feature). The numerical questionnaire for the first level of a decision model for selecting a retail store site is shown in Figure 2. A shortcoming of this feature is that questionnaires can only be output on a node-by-node basis. This presents a severe time imposition for large problems, since printing all questionnaires for an entire model (a typical need) requires a slow serial node-by-node manual process. During the product testing, it took more than 15 minutes to produce all questionnaires for a mid-sized model. (Note: As of April 1996, a change was made in the software that allows questionnaires and reports to be generated for any plex or the entire model with one command.)
Figure 2
Inconsistent expert judgment can be a factor when using the pairwise comparison method. The Evaluation and Choice module calculates and displays the Inconsistency Ratio (IR) of the AHP technique. The IR provides a measure of the logical rationality of the pairwise comparisons: a value less than 0.10 is generally considered acceptable. The Evaluation and Choice module calculates the IR, identifies the nine greatest inconsistent judgments (in ranked order) and recommends more consistent values.
The latter two features are under user control. Although very useful for detecting inconsistencies, these features should be used with caution, as the objective is to create a model of a physical process and not to create a perfectly consistent model. Moreover, during the product review, the system once recommended a value of 27.6 to improve consistency, well outside the 1-9 range of the pairwise comparison scale. (Note: The pairwise comparison mode can be extended from 1-99 in the numerical mode and in the graphical mode. The system is not recommending changing the value, but just reporting what value would provide the best fit.)
For some classes of criteria it may be more appropriate to directly enter data values. For example, assuming cost is a criterion then the actual values associated with each alternative are the best measure of each alternative's respective priority. As a standard feature, Expert Choice Pro allows direct data entry of values such as probabilities, sizes, dollars, lengths or any other values.
The ratings comparison mode is useful for applications involving large numbers of alternatives. Rather than comparing each alternative against specific criteria, standards are established for each criteria and prioritized (based on pairwise comparison between the standards). Alternatives are then compared against the standards of each criterion, e.g., excellent, above average, average and poor. The number of necessary pairwise comparison is thus a function of the number of criteria and not of alternatives. The ratings mode uses a spreadsheet format to facilitate data entry and review (data can also be imported or exported to a spreadsheet program).
After the decision model is completely defined (expert judgments made for all nodes and alternatives), the Evaluation and Choice module can be used to synthesize a solution to the model. The final result is an overall weight (or total score if the rating comparison mode is used) for each alternative. The results are presented in bar graph form, as shown in Figure 3.
Figure 3
A controversial aspect of the AHP technique is the possibility of rank reversal after modification to a model [Saaty, 1994; Canada and Sullivan, 1989]. Expert Choice Pro addresses this issue by providing two methods for synthesizing final results: a distributive method and an ideal method. The distributive method distributes the weights of the criteria among the alternatives at the particular level. Rank reversal can occur when using this method. Note, however, that some theorists consider rank reversals not only possible, but sometimes desirable.
The ideal method does not normalize the weights of the criteria, but assigns all the weight to the most preferred alternative. Rank reversal thus cannot occur since the most preferred alternative receives the entire priority, i.e., the ranks of the alternatives do not depend on each other. The software and the documentation provide guidance in the use of these methods, but care is needed to appropriately apply and interpret the results from these alternative methods.
The Evaluation and Choice module provides six standard reports on the principal aspects of a decision problem. These reports produce hard-copy output of the model hierarchy, expert assessments, synthesis results and user-entered comments. Reports may be copied to the Clipboard and imported into Windows applications as bit map pictures. Although the standard formats provide comprehensive coverage, report contents or formats are not user configurable.
Sensitivity Analysis
Sensitivity analysis examines the sensitivity of the results to changes in the priorities of the criteria. This is a particularly important aspect of an AHP problem analysis, since results are based on subjective expert assessments. Version 9.0 builds upon the excellent interactive sensitivity analysis tools of the previous release [Bahouth, 1993]. Sensitivity analysis can be performed from any level in the hierarchy; the software displays the sensitivity of alternatives to priority changes of the criteria immediately below a user selected node. This flexibility is very useful for fine tuning the sensitivity analysis.
The Evaluation and Choice module provides five different graphical modes for performing sensitivity analysis: 1. performance; 2. dynamic; 3. gradient; 4. 2-dimensional plot; and 5. differences.
Each of these graphical modes provide a different viewpoint to a sensitivity analysis. Under any of these five modes, the user can easily manipulate criterion priorities and immediately see the impact of the change (as reflected in the ranking of alternatives). Figure 4 illustrates the graphical interface of the performance sensitivity analysis as applied to a site selection problem. Here the sensitivity analysis is performed with respect to the goal node, level 0.
Figure 4
The terms VISIBLE, COMPET'N, CUST FIT and COST refer to the four nodes immediately below the goal. The left y-axis represents the relative priority of each criterion (as synthesized from the expert pairwise comparisons). The right y-axis represents the overall priority of each alternative (with the OVERALL axis showing the overall priority of each alternative). The vertical bars represent the derived relative priorities of each criterion.
Dragging any of the vertical bars causes immediate change in the priority of each alternative. According to the User's Manual, "The lines for the alternatives in between the vertical criterion lines have no meaning" [Expert Choice Pro, 1995, page 169]. The outputs for any combination of modes can be tiled so that they may be viewed simultaneously.
A variety of insightful sensitivity analysis studies can be performed using the standard features. It was difficult, however, to interpret the output of the 2-dimensional mode. The other four modes, by contrast, presented very useful information in a readily understood format.
Test Applications
As part of this software review, two groups of users were asked to apply Expert Choice Pro to actual multicriteria decision problems. The first application was a public policy problem for a city in southeastern Virginia. The problem was to decide among a number of alternative sports complexes in order to provide sufficient playing capacity over a 10-year planning horizon. A team of graduate students in the Department of Engineering Management at Old Dominion University was responsible for developing a model for the problem.
The second application was a personal decision problem: selecting among different school choices for our son. In this problem, my spouse and I developed the model for the problem. Prior to the start of the respective projects, the group of students and my spouse were familiar with decision analysis but had limited exposure to AHP, and no familiarity with Expert Choice Pro.
The combination of a well-designed human interface and the Online Tutorial allowed the users to quickly start using the software. Questions about specific details were easily answered through the Online HELP or the User's Manual. One student commented that the Online Tutorial was also useful as a periodic refresher (valuable for the occasional user). They both agreed that the judgment elicitation facilities and model creation tools were outstanding.
Asked to provide an overall assessment of the applicability of the tool and the ease of use of Expert Choice Pro, everyone agreed that it is an excellent product for addressing complex problems. However, I noted that although by the end of their work all participants were very comfortable with the fundamentals of the technique, there were uncertainties regarding some of the finer details of Expert Choice Pro. These first-time users were unsure regarding the suitability and interpretation of features such as the different assessment methods, the ideal and distributive synthesis modes, and the 2-dimensional plot sensitivity analysis. These uncertainties are understandable given that, in all cases, this was their first time use of the AHP. However, it does highlight the need for careful use of this powerful software product.
Overall Assessment
Expert Choice Professional Version 9.0 for Windows is an excellent product for addressing complex, multicriteria problems. Its design will satisfy both seasoned and relatively inexperienced analysts. Furthermore, Dr. Saaty's personal involvement with Expert Choice Pro ensures its theoretical soundness (with respect to the state of the art concerning AHP) and breadth of features.
Version 9.0 includes a revised User's Manual (with condensed quick reference card) and a new Online Tutorial (with accompanying hard copy manual). These provide generally very good descriptions of the software and its applicability to multicriteria problem solving. The Online Tutorial is particularly noteworthy for its effectiveness.
Any relatively minor shortcomings do not detract from the outstanding capabilities and features of Expert Choice Professional Version 9.0 for Windows.
References
1. Bahouth, S.B. (1993), "Expert Choice: Version 8 of Decision-making Software is Easier to Learn," OR/MS Today, Vol. 20, No. 3, pp. 62-65.
2. Canada, J.R. and Sullivan W.G. (1989), "Economic and Multiattribute Evaluation of Advanced Manufacturing Systems," Prentice-Hall, Englewood Cliffs, N.J.
3. Expert Choice Professional Version 9.0 for Windows, "User's Manual," Pittsburgh, Pa.
4. Saaty, T.L. (1994), "How to Make a Decision: The Analytic Hierarchy Process," Interfaces, Vol. 24, No. 6, pp. 19-43.
Abel A. Fernandez is an assistant professor in the Department of Engineering Management at Old Dominion University. His areas of teaching and research interests include project management, decision analysis and risk analysis.
Vendor & Product Information
Expert Choice Pro is produced and marketed by: Expert Choice Inc., 5001 Baum Boulevard, Suite 650, Pittsburgh, PA 15213; Phone: (412) 682-3844; Fax: (412) 682-7008.
The present pricing of Expert Choice Pro Version 9.0 for Windows is:
The trial and student versions have limited features and come with limited documentation. Special pricing is also available for volume orders, upgrades from previous releases and for site licenses.
Expert Choice Inc. also provides a number of other decision support software products, publications and seminars on decision analysis, and consulting services. Their home page on the World Wide Web provides up-to-date information on their products and services, location: http://www.expertchoice.com.
E-mail to the Editorial Department of OR/MS Today: orms@lionhrtpub.com
OR/MS Today copyright © 1997, 1998 by the Institute for Operations Research and the Management Sciences. All rights reserved.
Lionheart Publishing, Inc.
2555 Cumberland Parkway, Suite 299, Atlanta, GA 30339 USA
Phone: 770-431-0867 | Fax: 770-432-6969
E-mail: lpi@lionhrtpub.com
Web Site © Copyright 1997, 1998 by Lionheart Publishing, Inc. All rights reserved.
Web Design by Premier Web Designs, e-mail lionwebmaster@preweb.com