December 1997
It's Cool, But Is It Worthwhile?
Overwhelmed by information overload and by a seemingly continuous diet of meetings, a federal agency turns to information technology for answers. But does it help? Measuring the benefits of technological upgrades.
by Douglas A. Samuelson and Barbara Christoph
It's a familiar story: an organization introduces a major technological improvement. Recently it's the Internet; microcomputers, teleconferencing and word processors have had their turn, among others. The organization justifies the change in terms of improving business efficiency and effectiveness. The employees, or at least many of them, like the new technology and claim it's an improvement. Sooner or later, though, top management asks, "What benefit did we gain from this, really?" And more often than not, no one is comfortable trying to answer the question.
OR/MS professionals are often involved in such technological improvements. We are frequently among the early users, and in many cases we are among the advocates. Scanning the business literature, however, it is clear that we are much less often among those who take the lead in assessing what changes are worthwhile — and, consequently, in deciding which new changes are worth trying.
This disparity helps our competitors from other disciplines convince top management that OR/MS analysts would rather play with technology than help make real business decisions. It also represents a serious missed opportunity since OR/MS analysts are exceptionally well qualified to make good evaluations.
One Source of Measurement Error
Many analysts with more basic business backgrounds simply examine corporate profits before, during and after major technological upgrades. We've lost count of such articles which have appeared within the past few years; a Web search will easily turn up dozens. Almost invariably, these analysts find that profits are not greatly affected, with the possible exception of a drop in profit at the time the major investment payment is made. There is a vigorous discussion, therefore, of why technological upgrades which are almost universally praised, like the microcomputer, seem to have so little measurable impact on the bottom line.
The problem is that profit is a poor measure of actual business success. Taxes are based on profit, so prudent corporate managers try to keep profits positive (to keep investors happy), but modest. In a highly profitable year, companies seek to divert some profit into investments which will produce a short-term write-off (accounting loss, not necessarily involving actual expense) in exchange for long-term gains. In some cases, the company will make the investment even when the net present value of the expected gains may not greatly exceed the short-term loss, just for the ax-deferring and tax-avoiding benefits.
In short, companies manipulate their reported profits to reap tax and investment gains. Therefore, profits of companies which are doing well often stay fairly steady even when large real gain or losses occur. Because of this, if the most popular measurement methods were applied as uncritically to other topics as they are to technology, we would have to conclude that oil companies do not gain much by finding oil!
What managers do try to maximize is long-term growth of assets, or return on investment — and total net returns to top managers.(It is a naive analysis, indeed, which ignores the week-long management retreat in the Cayman Islands as part of actual executive compensation, but in accounting terms, it shows up only as a cost item in the training budget.)These criteria are much harder to measure, so they tend to be neglected. Of course, in non-profit and government organizations, the measurement problem is even more difficult, since the "bottom line" may be replaced by several criteria which need not be entirely consistent. Certainly one major criterion in this arena is an increase in productive labor hours.
Demonstrating a New Measurement Approach
So what can we do to determine whether technological changes have brought about real gains for the organization?
The answer is simple and obvious, but not easy to do: measure what actually did change, and evaluate what benefits the organization received from the changes.
As an example of this methodology, consider a current case study in which PUMA Systems, Inc., a small systems integration professional services and evaluation firm based in the northern Virginia suburbs of Washington, D.C., has the lead on the evaluation portion of the task. The client, a federal agency, has a number of work groups and project teams being supported, as part of a demonstration pilot project. To provide this support, the client chose a Web-based information management, collaboration and workflow tool (Livelink, by OpenText) and an electronic conferencing and information management tool (Group Systems by Ventana).
The agency's concern was that its people were overwhelmed by information overload and by a seemingly continuous diet of meetings. In a 16-week period, the team's analysts found that the 60-person group averaged 50 meetings per week, totaling 75 hours, which implies an average duration of about an hour and a half per meeting. Average attendance was not available, but many of the meetings the team observed had 10 to 20 people from various organizations all over the agency, and outside contractors in attendance. This sort of meetings load is consistent with findings from studies in industry for organizational units of comparable size. In addition, many of the meetings included one to three people by telephone conference call, and some of the meetings required overnight travel by a number of the participants.
The information management tool provides on-line capabilities to store, retrieve and search for documents; to post schedule meetings schedules; and to assign and track documents and workflow. It serves as an "intranet," providing a controlled method of information-gathering and processing capabilities within an organization that is geographically dispersed. In addition, it provides forums and discussions to assist members in preparing more effectively for meetings from their own homes or offices. From icon-oriented displays, users can move through a structured presentation of what documents are present and how they are related to one another.
The client hopes that many meetings will be avoided and problems resolved using the technology. The demonstration project is designed to support 500 to 1,000 users and is currently supporting more than 250, with new groups joining weekly.
As part of the demonstration project, many meetings will be supported electronically, with participants using laptop computers, to help collect and prioritize the meeting elements. When a topic is opened for discussion, participants can enter their comments simultaneously and read each other's comments more selectively and efficiently than they could process spoken comments in a meeting. In some cases, these meetings are assisted by trained facilitators.
Measuring Process Changes
To measure the process changes this demonstration will generate, the project team started by defining a baseline of information about the organization's processes and its meeting patterns. To do this, the team obtained meeting schedules and, where available, agendas and minutes. The team interviewed key personnel, using a structured protocol, to form assessments of which functional areas were critical to most people in the organization; and, for critical areas, what inputs the person needed (documents, databases and other source information), what outputs they generated, and who received those outputs. From this, the team developed an overall understanding of which activities and information resources seemed to be most important to the organization's mission.
The team used the meeting schedules and records, bolstered by some additional interviews, to estimate measures such as:
- proportion, number and expense of meetings held for information purposes only;
- proportion of meeting time expended providing information which participants require before they can address the decisions and actions the meeting was intended to address;
- resolution of action items and completion of assignments from meeting to meeting; and
- the extent to which information problems affect resolution of action items and completion of assignments.
- patterns of queuing for resources, such as on-line and document servers;
- throughput of information requests;
- number of assignments made through the system;
- number of tracking requests made through the system; and
- number of document searches processed through the system.
These measures make it possible not only to analyze changing patterns of system use, but also to estimate capacity requirements: whenever significant queues form for a resource, and especially when such queueing coincides with increased user complaints about slow response or difficulty in getting tasks done, the load measures at that point indicate the effective capacity of the resource. Thus, for example, if a document server begins to show unacceptably slow response when there are 60 or more search inquiries per hour, then we conclude that 60 inquiries per hour is the effective capacity of that type of server for this type of application. What the team has already seen is that some capabilities are most affected by the number of users logged on at once, others by the numbers of certain types of inquiries, and still others by the volume of information users attempt to download simultaneously.
Measuring Performance
Finally, the team is collecting such performance measures as time and resources expended:
- in meetings;
- searching for needed information;
- to accomplish critical tasks;
- to make assignments and monitor work flow;
- on routine reports and tracking status;
- keeping and retrieving records of meeting results;
- on interruptions generated by needs for information;
- retrieving lost information.
Comparing these measures over the time before and during the demonstration project, the team will be able to estimate the benefits the organization derived from the new tool. These benefits could be considerable. As we noted earlier, the organization is averaging about 75 hours of meetings per week, and the team estimates that, on average, at least 10 people attend these meetings. If the organization's staff members' time is valued at $100 per hour, which reflects the usual multiplier of 2.5 to 3.0 times base salary to account for benefits, leave, facilities and overhead, then the organization's 750 person-hours of meetings are worth $75,000 per week, or about $3.9 million per year. If as little as 10 percent of this time is wasted and can be reclaimed for productive activity, the benefit to the organization would be $390,000 per year from the labor cost alone.
Since more than half the meetings the team observed seemed to be solely for the purpose of disseminating information and making sure that the participants had received it, the team considers it more likely that perhaps 30 to 40 percent of the organization's meeting time is currently wasted. If most of this time could be reclaimed for good use, the organization's benefit would be more than $1 million per year.
This project illustrates both the potential benefits and the measurement challenges associated with information technology improvements in large organizations. The project team looks forward to reporting more results in six months to a year. Meanwhile, the measurement approaches and specific measures the team has developed serve as a model for other OR/MS analysts who want to do a better job of evaluating the benefits of changes in how organizations do business.
Douglas A, Samuelson, a frequent contributor to OR/MS Today, is Principal Scientist for PUMA Systems. Barbara Christoph is President of PUMA Systems.
OR/MS Today copyright © 1997 by the Institute for Operations Research and the Management Sciences. All rights reserved.
Lionheart Publishing, Inc.
506 Roswell Street, Suite 220, Marietta, GA 30060, USA
Phone: 770-431-0867 | Fax: 770-432-6969
E-mail: lpi@lionhrtpub.com
Web Site © Copyright 1997, 1998 by Lionheart Publishing, Inc. All rights reserved.