by
Beverly E. Thomas, John P. Baron, and Wayne J. Schmidt
Many Army personnel can be classified as knowledge workers-people who produce not tangible products, but some form of processed or enhanced information, often using processes that allow a high degree of individual discretion in task performance. Knowledge work is the area that offers the greatest opportunity to increase productivity within the U.S. workforce. Ongoing research at the U.S. Army Construction Engineering Research Laboratories (USACERL) is developing the Knowledge Worker System (KWS), an integrated performance support environment (PSE) designed to improve the performance of Army knowledge workers.
KWS promises to offer significant benefits. However, before installing any new system, a prospective user must evaluate whether the benefits of installation outweigh the costs, in terms of both time and resources. This study undertook to identify appropriate methods to evaluate the feasibility of implementing or continuing to use a PSE for knowledge workers, and concluded that a "toolkit" of five evaluation techniques, each applicable to a specific workgroup setting, may best assess the feasibility and usefulness of a PSE:
- Work Profile Analysis
- Direct to Indirect Ratio
- Time Saved Times Salary (TSTS)
- Activity Based Costing (ABC)
- Quality Assessment.
Back to KWS Home Page
Foreword
This study was conducted for the Directorate of Military Programs,
Headquarters, U.S. Army Corps of Engineers (HQUSACE) under Project
4A162784AT41, "Military Facilities Engineering Technology";
Work Unit FFAJ5, "Performance Support Environment Effectiveness."
The technical monitor was Erica Ellis, CEMPMC.
The work was performed by the Business Processes Division (PL-B)
of the Planning and Management Laboratory (PL), U.S. Army Construction
Engineering Research Laboratories (USACERL). Moonja P. Kim is
Acting Chief, CECER-PL-B, L. Michael Golish is Acting Operations
Chief, and David M. Joncich is Chief, CECER-PL. The USACERL technical
editor was William J. Wolfe, Technical Resources Center.
COL James T. Scott is Commander and Acting Director of USACERL,
and Dr. Michael J. O'Connor is Technical Director.
1 Introduction
Background
Many Army personnel can be classified as knowledge workers-people
who produce not tangible products, but some form of processed
or enhanced information, often using processes that allow a high
degree of individual discretion in task performance. Knowledge
workers make decisions that significantly impact organizational
resources and are themselves a significant and costly resource;
knowledge workers compose 43 percent of the white-collar sector,
which in turn comprises 67 percent of the service sector (Roach
1991).
Knowledge work is the area that offers the greatest opportunity
to increase productivity within the U.S. workforce (Drucker 1974).
Ongoing research at the U.S. Army Construction Engineering Research
Laboratories (USACERL) is developing an integrated software program
designed to improve the performance of Army knowledge workers.
The Knowledge Worker System (KWS) is a computer-based performance
support environment (PSE) designed to document, coordinate, and
execute the business processes assigned to workgroups. KWS is
an integrated set of automation systems that guides Army action
officers through the course of their daily tasks by helping them
organize, prioritize, and execute their work efficiently and effectively.
KWS can help meet the need for a wide range of functional process
improvements and training requirements by providing institutional
knowledge on an as-needed, individualized basis.
While KWS promises to offer significant benefits, installing any
new system involves a commitment of both time and resources that
must justify itself economically; i.e., benefits must outweigh
costs. Part of the KWS research effort is to identify appropriate
metrics that indicate the impact of the system on workgroups.
This study undertook to investigate techniques to measure whether
the implementation and/or continued use of a PSE for knowledge
workers such as KWS is justified.
Objectives
The objective of this research is to identify appropriate methods
to evaluate the feasibility of implementing, or continuing the
use of, a performance support environ-ment for knowledge workers.
Approach
A broad review of literature in productivity measurement was conducted,
including areas such as white collar productivity, organizational
psychology, information economics, industrial engineering, economics
of technology, quality management, and decision theory. Several
techniques for productivity measurement were evaluated for their
applicability to the environments in which Army knowledge workers
operate.
Mode of Technology Transfer
The findings from this study will be incorporated into future
USACERL work units that address the development of the Knowledge
Worker System.
2 Issues in Evaluation
"What gets measured gets attention, particularly when rewards
are tied to measures."
Why Measure
"Productivity" is a fairly intuitive notion that can
be defined as the relationship of the outputs to the inputs used
in production (National Research Council 1994). If outputs and
inputs are well quantified, that relationship might simply be
defined-and compared-mathematically, as output divided by input
(Sink 1984).
The goal of productivity measurement is to determine whether organizations
can obtain the same output with fewer resources, or increase output
while holding resource levels constant (Lau 1988). With this goal
in mind, three reasons for evaluating productivity are:
Within the Federal government, the Bureau of Labor Statistics
monitors productivity by collecting output measures, identifying
resource requirements, and determining estimated production goals
(Forte 1992). Objective measures of output, resource levels, and
production goals are benchmarks that form a baseline to compare
and improve productivity levels. Without such benchmarks, organizations
could not set goals or allocate resources since, "without
productivity objectives, a business does not have direction. Without
productivity measurement, it does not have control" (Drucker
1974). It is this baseline of knowledge worker productivity that
must be determined (Bridges 1992).
Problems in Evaluating Knowledge Work
Measurement and evaluation of knowledge work is a difficult problem
(Drucker 1974; Magliola-Zoch 1984; Rittenhouse 1992; Sassone 1991;
Sink 1985; Thomas and Baron 1994). Many of the problems relate
to the very nature of knowledge work: its inputs are not clearly
definable; it generates intangible outputs; and it allows a high
degree of discretion on the part of the performer (Beruvides and
Sumath 1987). Knowledge work is often complex and nonroutine,
and commonly requires the contribution of several people to complete
a given task. All these characteristics make norms and standards
difficult to establish, and performance hard to measure.
The net results of this complexity is disagreement about what
to evaluate. Frequently, inappropriate metrics are applied to
knowledge work simply because the particular indicator in use
is easy to quantify (Rittenhouse 1992). The essential component
of quality is often ignored due to the difficulty in measuring
this attribute.
In 1994, the National Research Council commissioned a committee
to investigate the apparent meager increases in productivity in
the service sector, despite the large amounts of money spent on
information technology for this group. The committee concluded
that, to a large degree, these disappointing productivity results
arise from inappropriate metrics:
Productivity data do not capture important elements of service
output. Key among these are the capacities to handle increased
complexity and to provide improved timeliness, flexibility, response
times, reliability, or safety for employees, customers, or the
general public.
If these and other factors are accounted for, the resulting figures
might demonstrate a strong increase in service-sector performance
through information technology investments (Peterson 1994, p 7).
Individual vs. Group Measures
Rittenhouse (1992) and Sassone (1991) identify the workgroup as
the appropriate level at which to measure the performance of knowledge
workers. Rittenhouse observes that individual measurements are
not particularly useful since increases in an individual's productivity
does not necessarily transfer upward within an organization. For
this reason, this study concentrated on group evaluation.
Several practicioners recommend the use of a family of measures.
The director of quality measurement and improvement at USAA has
credited the ability to track both individual performance (in
some categories) and group performance (in others) as a factor
in the organization's finalist position in the Malcolm Baldridge
National Quality Award for multiple years (Helton 1992).
Helton (1992) recommends an approach that includes the following
steps: (1) select the group involved in the work to be tracked,
(2) help this group select several measures appropriate to the
work, (3) help the group clearly define the measures, frequency
of measurement, and whether benchmarking is appropriate, and,
(4) document the results.
3 Evaluation Strategy
Suggested Approach
In earlier work, Thomas and Baron (1994) concluded that knowledge
work performance measurement required special evaluation techniques.
No single technique is appropriate for all types of knowledge
work. The authors suggested that the first step in performance
measurement within knowledge work environments was work categorization.
Thomas and Baron identified eight components of knowledge work
relevant to categorizing work groups:
This earlier work suggests that a "toolkit" of evaluation
techniques is appropriate to evaluate the feasibility of a performance
support environment for knowledge workers. Industrial engineering,
industrial management, and operations research have yielded a
variety of evaluation techniques, each applicable to specific
work group settings.
Before determining which techniques fit a particular work group,
the first step should be to categorize the type of knowledge work
performed within that environment. Reducing the eight work components
to four key components reduces the categorization effort, while
still maintaining clear and useful categories:
The remaining two components, Structure and Skill Level,
can be eliminated from work categorization exercise without sacrificing
meaningful results since structure appears to vary significantly
within knowledge work environments and skill level is more relevant
to blue-collar work environments than to knowledge work.
Helton (1992) suggests four criteria for the categorization of
knowledge work:
Helton uses these criteria to classify work into one of four types
of white-collar work: Specialist, Professional, Support Staff,
and Clerical Staff (Table 1). Helton's categories correspond roughly
with the suggested framework: Work Range correlates with Time
per Job; (2) Work Structure correlates with Volume and Repetitions;
(3) Controls correspond with Structure, and (4) Cognitive Effort
correlates with Complexity.
Selection of Evaluation Techniques
Table 2 and Figure
1 show
how work is
categorized by
four work
components to
identify three
resulting types
of work
environments:
Knowledge Work,
Production
Office, and
Proceduralized
Work (formerly
called blue
collar work).
Each of these
environments
exhibits a range
of values for
each of the four
work components.
Proceduralized
(blue collar)
Work is included
only as a point
of comparison
with the two
knowledge work
environments.
Once a work group has been categorized by the four composite knowledge
work attributes, the next step is to choose an appropriate evaluation
tool. The workgroup itself must participate in establishing the
particular metrics to be used for evaluation. Besides being the
best judges of appropriate metrics, the more involved the group
is, the less its members are likely to feel threatened by the
study (Anthony 1984; Bernard 1986).
Information Economics
Economists Parker, Benson, and Trainor (1988) proposed Information
Economics (IE) as a framework that quantifies and weights intangible
benefits and costs of information technology (IT) alternatives.
IE is intended as an approach for evaluating an organization's
IT investments.
Standard cost-benefit analysis, the conventional tool for evaluating
IT projects, is based on accounting-type data such as projected
costs and benefits, and estimated return on investment (ROI).
However, this standard approach ignores a number of factors that
may determine the success or failure of a project simply because
these factors do not lend themselves to dollar quantification.
Traditional cost-benefit analysis is not adequate for evaluation
of innovative systems such as performance support environments
(Parker, Benson, and Trainor 1988).
Information Economics can be defined as a collection of computational
tools that allow rational comparison of benefits and costs of
IT projects. Parker, Benson, and Trainor claim that IE goes beyond
cost-benefit analysis by providing a decisionmaking structure
that separates information technology justification from technology
feasibility for a particular project. In other words, IE allows
an organization to make two important distinctions: (1) what this
project is worth to the organization, and (2) whether the organization
has the resources necessary to complete the project. In terms
of justifying a PSE, Information Economics is valuable mainly
because of the tools it provides for determining an answer to
the first question.
Figure 2 shows how IE focuses on value rather than the more limited
concept of benefit and hard dollar savings. The model expands
the traditional cost-benefit analysis to include quantification
of intangible benefits and risks of both business and technical
issues. Dollar savings are important, but intangible elements
such as increased knowledge worker productivity, improved communications,
and enhanced quality should be evaluated by organizations as they
make IT decisions. IE typically suggests six classes of value:
Each of these value classes is assigned a weight for the organization
in which the IE analysis is occurring. Eventually, every proposed
project will be evaluated in each value class for its effect and
then a summarized score is calculated for each project.
In applying IE concepts in a cost-benefit analysis, an organization
typically begins from its business vision and then establishes
criteria and weights that can be used in deciding which IT investments
best fit the vision. When applying IE to the justification of
a PSE, the starting point is a set of potential benefits that
can be attributed to implementation of the PSE. Consider the following
example:
PERFECT Company places great value on never making a mistake on
a customer's order. Their vision is focused on this error-free
goal. Their evaluation of any project will weigh heavily its effect
on perfection. CHEAP Company on the other hand takes great pride
in offering the cheapest price on the market. Consequently, this
company will weigh any project heavily for its effect on price.
This example shows that what the user deems important must be
established as a first step in evaluation. Once an evaluation
team has determined what the user finds important in a potential
IT project, the selection of appropriate tools and measures will
follow. In the example, the overriding concern of the PERFECT
Company is perfection. Logically, the business will look for measures
that reflect the level of perfection.
Suppose IE is applied to the justification of KWS. One of the
benefits of KWS is an improvement in scheduling efficiency. If
one member of the evaluation team rates this benefit as very important,
then one of the metrics for determining KWS success for this rater
will be the improvement of scheduling. A point central to the
IE concept is that justification of IT is based on factors important
to the IT user.
There are four steps in performing the IE-derived cost-benefit
process:
The application of IE cost-benefit analysis to the evaluation
of a performance support environment follows the same four-step
procedure. The group that performs this analysis should be comprised
of representatives of top management and personnel who will primarily
use the PSE.
The value classes supported by the PSE must be identified. The
following classes are proposed:
The major benefits of the PSE must be clearly identified. The
key KWS benefits are:
The worksheet shown in Figure 3 reflects these proposed value
classes and the KWS benefits. The key point for the justification
of a PSE using the IE concept of cost-benefit analysis is to perform
this type of analysis at the beginning of the project. Use the
results of the IE analysis to: (1) gain consensus among management
and primary users about the expected benefits, (2) influence the
choice of performance metrics and, (3) create an audit trail of
the information and decisions that initiated the PSE.
Appendix A provides a taxonomy of KWS benefits, capabilities,
and functions as well as an explanation of the worksheet example
used here. Appendix B gives more information on Information Economics.
4 Toolkit of Evaluation Techniques
Several techniques should be included in a toolkit for evaluating
the feasibility of implementing a PSE for knowledge workers. Each
technique is described here.
Work Profile Analysis
Work Profile Analysis is part of the hedonic wage model, as applied
to the justification of an office information system (OIS) by
Peter Sassone (1987, 1992, 1992). This application of the hedonic
wage model assumes that an OIS can both decrease the amount of
time required to complete a given task and facilitate restructuring
of work assignments. Both of these changes are postulated to result
in higher efficiency. Professionals have more time to perform
work in their specific specialties and spend less time on routine
and/or nonproductive tasks. The combination of the OIS and restructuring
within an office can correct the misallocation of time spent by
white-collar professionals on lower-value activities.
The premise of the hedonic wage model, as applied to OIS, is that
the value of an information system originates in the value of
the activities performed by the intended OIS users and how the
target system improves work patterns. This premise matches the
purposes of a PSE for knowledge workers and applies to the task
of evaluating the impact of a PSE.
A key component of the Sassone's application of the hedonic wage
model is the Work Profile Analysis (also referred to as Work Value
Analysis), which is based on the concept of intellectual specialization
within knowledge work. The approach begins with a baseline analysis
of work patterns in an office before introduction of an OIS. After
the OIS is implemented, work patterns are resurveyed. Comparing
the "before" and "after" work pattern snapshots
provide the basis for identifying the impact of the system.
Sassone explains the concept of intellectual specialization as
follows. Organizations generally pay their personnel on the basis
of the intellectual content of the work each employee is capable
of doing. Personnel with great experience are compensated at a
higher rate than inexperienced personnel. Likewise, employees
with advanced degrees are paid more than those with lesser degrees.
Engineers, for example, are paid more than secretaries.
Consider, however, that employees do not spend 100 percent of
their work time doing the kind of work their background qualifies
them to perform. Knowledge workers may spend only a small portion
of a work day working in their area of expertise. Most knowledge
workers exert much of their work effort doing tasks that could
be delegated to less skilled, less expensive employees (Sassone
1992).
A Work Profile Analysis can indicate how knowledge workers at
various levels within an organization typically spend their time.
By calculating the typical number of hours worked and the total
cost of each type of work, the actual cost of each type of knowledge
work within a specific organization can be figured.
The steps in creating a Work Profile Analysis are to: (1) categorize
the work, (2) survey the employees, (3) develop a matrix analysis,
(4) implement the system under study, (5) resurvey employee, (6)
compile a second matrix analysis, and (7) compare the baseline
and second matrices to evaluate the impact of the information
system.
The purpose of work categorization is to classify work performed
by specific workgroups. This phase is accomplished by reviewing
mission statements, job descriptions, and interviewing (both management
and nonmanagement) personnel. The workgroup's tasks are decomposed
to the lowest level to which each can be delegated. Tasks are
then consolidated to produce an averaged grouping of functions.
The output of this step is a survey instrument.
In the survey step, each workgroup member logs data that indicates
how much time he or she spends within each category of work. In
Sassone's studies, each knowledge worker recorded information
hourly in response to the question: "How many minutes did
you spend in each of the pre-defined categories of work?"
A matrix analysis is created by gathering data from the surveys
and compiling the information into a matrix that indicates the
workgroup's intellectual work distribution by organization position.
This matrix will serve as a baseline for the workgroup under study.
For example, the information gathered from the surveys (and initial
interviews) is compiled into a preliminary matrix (Table 3) that
shows distribution of effort within each staffing level. This
information will serve as a baseline for the workgroup under study.
Matrix Analysis I
Table 3 shows that managers spend 30 percent of their time performing
managerial work, (e.g., planning, personnel work, budgeting, upward
reporting), 20 percent of their time in professional work (e.g.,
preparing technical presentations, writing technical reports,
technical analysis), 20 percent of their time in support work
in support tasks (e.g., clerical tasks, fixing hardware or software,
filing, photocopying, or keying in data). The remaining 30 percent
of their time is spent on nonproductive tasks (e.g., looking for
information, waiting on people or equipment, playing telephone
tag, walking between buildings).
Following Sassone's approach, the office information system is
implemented and allowed to operate long enough that all start-up
effects are over. Once the system is in its operational stage
and system benefits can be realized, the workgroup is resurveyed.
Results of this second survey are compiled into the second matrix.
Employees are resurveyed at that time to produce a post-implementation
matrix analysis.
Matrix Analysis II
The second matrix (Table
Table 4)
shows that
the overall
distribution
of time
within each
category of
work has
changed. The
implication
is that
managers
accomplish
more
higher-level
functions-the
desired
effect.
Assuming that
managers and
senior
professionals
are paid
higher
salaries than
support
personnel,
the
organization
can realize
an economic
gain from
this
shift.
The results of the Work Profile Analysis can be used to evaluate
the impact of the OIS. The results may also indicate the need
for additional support personnel or other restructuring within
the office. As Sassone has written, many organizations have managers
and highly-skilled professionals who spend too little time in
work requiring their expertise. Often, much of the professional's
time is spent in work that could be delegated to lower-paid employees.
In many cases, restructuring the office or workgroup can correct
this problem.
Specifically, there are often insufficient numbers of secretaries
and/or clerks to handle the volume of clerical work required.
Hiring additional clerical help or otherwise restructuring the
environment is usually a less expensive alternative than paying
higher-skilled workers to perform these tasks (Sassone 1992).
Work Profile Analysis can be accomplished in a PSE via an automated
or manual tool. USACERL has created a software instrument [Download Time Logger Software] that
achieves the time logging phases of the methodology (summarized
in Appendix C). The other steps, work categorization and development
of the work matrix, must be performed manually. Alternatively,
the entire process can be performed manually, as in Sassone's
studies (Sassone 1992). Appendix D to this report gives instructions
and a sample dictionary of work categories derived for knowledge
workers at an office in Headquarters, U.S. Army Corps of Engineers
(HQUSACE).
Computation of Direct to Indirect Work Ratio
A second possible approach to evaluating the impact of a PSE computes
the amount of time spent on direct vs. indirect work. Direct work
is defined as those activities required to generate mission-related
products. Indirect work includes tasks that support personnel
performing mission-related work. The concept of direct versus
indirect work originated in manufacturing. In industry, direct
work denotes production labor; indirect work refers to labor that
supports production. In a factory setting, an employee's work
is considered as either 100 percent direct or indirect labor.
If a worker is in operations or production, that employee's work
is classified as direct. Maintenance or clerical work is automatically
considered indirect.
The distinction between indirect and direct is more difficult
in knowledge work. White-collar professionals can perform both
types of tasks. In a knowledge work setting, indirect work includes
activities such as photocopying, upward reporting, personnel management,
filing, and responding to requests for information. Both classifications
of work are necessary, but the organization is best served if
most of a knowledge worker's time is spent on direct work.
Since knowledge workers may create goods or services, the same
employees are often responsible for many indirect assignments,
such as upward reporting, paperwork, distribution of information,
and personnel management tasks. Each workgroup within an organization
must find the appropriate balance between direct and indirect
work (Helton 1993).
One advocate of this approach suggests that knowledge workers
should strive to spend approximately 60 percent of their time
in direct work. Helton recommends performing organizational alignment,
a process in which the mission or purpose of the organization
is closely coupled with how work time is spent. He suggests that
employees who spend more time doing value-added work benefit both
the organization and themselves. Measures that focus work effort
and expedite work performance can result in continuous improvement
within the organization (Helton 1991).
Helton's approach would track several trends: (1) the amount of
direct work time, (2) the amount of indirect work time, (3) the
ratio of direct vs. indirect work time, and (4) the ratio of direct
to total work time. Comparison of the ratio of direct vs. indirect
work prior to and following the introduction of a PSE may yield
a useful indicator of the system's effectiveness.
Helton's approach does not emphasize the industrial engineering
focus on measuring activities, but rather concentrates on direct
work and speeding up work processes. He states that direct work
adds value to an organization by expediting the accomplishment
of the business-related, tangible outputs. By emphasizing direct
work and therefore the mission of the organization, knowledge
workers are encouraged to spend their time and effort on strategic
business objectives.
To apply these concepts to the evaluation of a PSE, the implementation
team that collects process information prior to introduction of
a PSE can be tasked to identify situations where direct versus
indirect measures are easily discernible. A workgroup with one
major purpose is a likely candidate for calculation of direct-indirect
work ratios.
For example, an organization's travel office has a clear mission:
to make travel arrangements for organizational personnel. If the
travel office agrees that the number of travel orders processed
quarterly is a good measure of their productivity, this data should
be targeted for pre- and post-implementation collection. The implementation
team should identify all tasks that pertain to the production
of travel orders, which will serve as the basis of the direct
work figure. The team must ascertain how many staff hours are
spent on work related to travel orders before the introduction
of the PSE. Any staff time spent on work not related to the production
of travel orders is figured as indirect work. The direct versus
indirect work ratio can be calculated as the percentage of staff
time spent on travel order production relative to the percentage
of staff time spent on other work.
A PSE installation generally requires an implementation team that
is responsible for collecting process information for the site.
The implementation team should examine tasks that comprise indirect
work as possible Process Improvement Possibilities (PIPs). Indirect
work tasks may be PIP candidates that should be eliminated, automated,
streamlined, or subjected to other process re-engineering activities.
After identifying direct work tasks and collecting the number
of hours spent on direct work, calculate the percentage of direct
work time (staff hours spent on direct work divided by total staff
hours). Likewise, figure the indirect work percentage (staff hours
spent on indirect work divided by total staff hours). A direct
to indirect ratio of 1.5 to 1 has been suggested as a reasonable
target for most professional organizations (Helton 1991).
The post-implementation measurement is taken after the PSE is
fully implemented. After the staff has been completely trained
and the start-up costs of initiating a new system have been realized,
the direct vs. indirect work ratio should be recalculated to determine
if the ratio has improved.
Time Saved Times Salary
The Time Saved Times Salary (TSTS) technique examines labor efficiency
gains that can be attributed to the implementation of a PSE. The
approach is straightforward; gains in efficiency are multiplied
by labor cost.
For example, if a task that took 4 hours to perform before implementing
a PSE can completed in 1 hour after system implementation, then
a time savings of 3 hours per task performance can be calculated.
If the task is required monthly, time savings of 3 hours per month
or 36 hours yearly is realized. To calculate an annual TSTS figure,
36 hours is multiplied by the hourly salary of the knowledge worker
who normally performs the task.
Estimates of task duration must be recorded before system implementation
for the TSTS technique. Documentation of task durations should
be the responsibility of the implementation team that collects
process information. It is important to ask personnel who actually
do the task how long performance takes. Ask task performers to
differentiate between elapsed time and time actually spent on
the task, from start-to-finish. A change in task performance can
affect either or both figures. Interview personnel to determine
volume (the number of times a targeted activity occurs
within a given time frame) and repetitions (the frequency
with which a process is performed). These questions must be asked
prior to system implementation to get accurate task durations.
Once the task has been automated or the process streamlined, any
estimates of duration will be less reliable. Intuitively, the
duration of a task is easier to recall before changing the task.
An additional application of this data addresses the selection
of PIPs. Tasks that have long durations, frequent iterations,
and/or high volume are high-priority candidates for PIPs. These
are tasks that may yield a high pay-off in efficiency gains and
therefore should be examined for automation, streamlining, or
other process improvements.
The final step in TSTS is taken after the PSE is fully implemented.
Some time following the introduction of the PSE, after the start-up
costs of initiating a new system have been realized, post-implementation
task performance data must be collected. Tasks that were targeted
as PIPs should be examined to determine if (1) duration, (2) frequency,
and/or (3) volume have changed following the PSE implementation.
These data can be collected via interview or survey. Alternatively,
the PSE can be constructed to collect the required data.
Activity Based Costing
Activity Based Costing (ABC) is a cost accounting method that
attempts to allocate the actual cost of providing a service or
producing a product. ABC differs from traditional accounting practices
that allocate all indirect costs through somewhat arbitrary accounting
rules. Traditional accounting techniques generally link all overhead
costs to products or services on the basis of direct labor costs.
According to conventional accounting rules, a service that generates
10 percent of the total direct labor costs for an organization
would also be allocated 10 percent of all overhead costs. ABC
attempts to allocate costs to the services or products that generate
these costs. ABC was derived from a manufacturing model that defines
production as a set of predefined activities. The activities consume
resources, which generate costs. By allocating a product (or service)
to a set of activities with an incumbent set of resources and
costs, a realistic cost of generating the product or service can
be calculated.
The following example illustrates the ABC approach (Liggett, Trevino,
and Lavelle (1992). Consider a company that produces two products:
gadgets and widgets. Four employees are responsible for performing
all work. Two employees spend 100 percent of their time in assembling
the components required to produce the gadgets and widgets. The
other two employees spend 70 percent of their time inspecting
the component parts and finished products, and 30 percent of their
time in material handling.
Each employee costs the company $10 per hour, including all fringes.
During the course of 1 year, 1.3 million components are received
and assembled into finished products. One million tests are performed
to inspect the parts and finished products. Components are moved
from storage into the assembly stations in batches of 50. The
distance traveled for retrieval of each gadget component is 250
ft; for widgets, the distance traveled is 40 ft (1 ft = 0.305
m). Gadgets require four components; widgets require six. Seven
tests are conducted for each gadget produced; two tests are conducted
for each widget. Annual gadget production is 100,000 units; annual
widget production is 150,000 units.
Using this data, unit cost values for gadgets and widgets can
be constructed using the ABC paradigm. There are three activities
in producing the two products: assembly, inspection, and material
handling. The cost drivers for each activity are: (1) assembly
- number of components assembled, (2) inspection - number of tests
conducted, and (3) materials handling - number of feet the components
are moved.
Tables 5 through 9 give a summary analysis of the process. Table
5 links the products to the activities through cost drivers. Table
6 links activities to resources and the resources to costs. Table
7 provides the calculation of the unit cost of each activity.
For example, the average cost of assembling components is $40,000
divided by 1.3 million, or $0.0308 each. Table 8 shows the unit
cost of producing each component. For example, the unit cost of
producing a gadget is $0.4073. Table 9 allows a comparison of
unit costs as calculated using ABC with unit costs as figured
using traditional cost accounting rules.
Table 6. Activities related to resources and costs.
Table 7. Determining the unit cost of activities.
Table 8. Calculating the unit cost of production via ABC.
Table 9. Calculating unit costs of production using conventional cost accounting.
In the latter case, the assembly costs of gadgets and widgets
are viewed as direct costs; the inspection and material handling
costs are treated as indirect. The indirect costs would be allocated
to gadgets and widgets in proportion to the direct costs of each
product. The amounts that would result from the conventional cost
accounting approach are calculated in Table 9. Note that the last
column indicates traditional cost accounting and yields an error
of approximately 40 percent in allocating the unit costs of the
two products.
ABC has been applied to the service industry in an effort to give
managers a framework for making sound business decisions by identifying
all the costs associated with providing a particular service.
Using ABC, organizations can more clearly see the true costs of
products and services. Organizations can use this information
to make decisions that improve the profitability of their operations.
ABC can be applied to the cost justification of a performance
support environment in certain specific situations. The ABC approach
can be useful in production offices and in structured knowledge
work environments where tasks are essentially repetitive.
Consider a production office where the primary work requirement
involves repetitively performing certain processes, such as preparing
reports, preparing budgets, and collecting data from other personnel.
The tasks may be repeated on a daily, weekly, monthly, yearly,
or other periodic basis.
Each process is initiated
by a specific
driver, such
as a request
from
management,
the approach
of the due
date for a
weekly
report, a
data call, or
appointment
to a
committee.
Each process
is composed
of tasks,
such as
entering data
into a
spreadsheet,
sending email
messages,
attending
meetings,
preparing
documents,
scheduling
meetings, or
making phone
calls. For
example,
Table 10 shows
the
application
of the ABC
technique to
the
preparation
of a weekly
report that
indicates how
60 knowledge
workers
charged their
time to
projects.
The ABC breakdown of processes, tasks, and costs provides information
that would be useful both in building the database of processes
and in selecting PIPs. However, the cost of the tasks that comprise
each process is difficult and expensive to assess. A manual or
automated time logging instrument could be constructed to help
gather the implicit costs of each task.
The time and expense of collecting this information, however,
is probably not warranted unless the workgroup being analyzed
is both structured and stable, i.e., if the tasks are always done
using the same procedures, and if the processes are not likely
to be changed for several months. If these conditions are not
met, the resources necessary to implement ABC are likely to be
greater than the benefit gained.
ABC is also recommended as an evaluation technique in those cases
where both Integrated Definition (IDEF) process modeling and ABC
have been performed as part of other initiatives. The Department
of Defense (DOD) has recommended the use of both IDEF and ABC
as part Functional Process Improvement efforts in the Defense
Information Management Program (1993).
If ABC has been implemented for workgroups where a performance
support environment (PSE) is being introduced, the results can
be used to identify the processes that consume the largest amount
of labor resources. The most costly processes should be examined
as likely PIPs, as mentioned previously. These same processes
may warrant further data collection after the PSE has been fully
implemented. This post-implementation data can be done using either
another technique such as Work Profile Analysis or Time Saved
Times Salary, or by re-application of ABC.
Quality Assessment
Improved quality, along with enhanced employee performance, is
a key benefit acclaimed by performance support systems (Gery 1991).
Along with any performance impacts an organization tracks, changes
in quality should also be monitored. The desire for improved quality
is one of the top six reasons that organizations invest in information
technology, where the top six reasons for investing in information
technology are to:
Paradoxically, organizations report difficulty in measuring the
impact of IT on quality. Further, the impact of the technology
investment is largely meaningless without a valid metric of the
quality of the resulting output (National Research Council 1994).
Service-oriented organizations have provided leadership in developing
tools and methods that track customer-oriented measures of quality.
These tools were developed in response to the strong positive
correlation between costs and the quality of service. Reduction
of errors in producing a service reduces both coordination costs
and rework, as well as customer complaints.
Organizations implementing a PSE are encouraged to develop and
use customer-oriented quality metrics for similar reasons. The
tools used by private sector service organizations are largely
applicable for use within government environments. For example,
some methods used by service organizations are: focus groups,
user groups, quality circles, process action teams, pilot tests,
surveys, sampling, interviews, observation, and other quality
management tools (Deming 1986). Each of these may be useful in
assessing quality within a specific setting.
Minimally, methods should: (1) collect customer feedback, (2)
collect supervisory feedback, and (3) provide opportunities for
self-assessment. This information can be collected fairly inexpensively,
using methods such as interviews, surveys, and/or tools built
into the system itself.
For example, KWS will use surveys and on-line tools to collect
quality-related data. Customers of KWS users will be surveyed
to ascertain their level of satisfaction with the timeliness,
completeness, and general quality of the products generated using
KWS. KWS users will be surveyed via periodic electronic mail surveys
about their own productivity. Supervisors of KWS users will likewise
be surveyed to assess the productivity and quality of work performed
by their subordinates who use the system. Finally, KWS users will
have the capability to summon an on-line tool that allows comments
on the usefulness of specific KWS features.
The information collected will be used to: (1) assess the perceived
quality of the work performed by KWS users, and (2) gather information
on the perceived usefulness of system functionality. The first
category of information will be collected prior to and following
KWS implementation.
Tools for collecting data about quality issues have been thoroughly
detailed in the Total Quality Management (TQM) literature (Brassard
1988, Deming 1986, Cleary 1993). Figure 4 lists several tools
described in TQM literature. The tools are grouped into "problem
identification" or "problem analysis" categories.
Several tools are useful for both types of problem solving.
Organizations implementing a PSE are encouraged to assess quality
along with the particular performance measures they choose to
track. Quality-related data from customers, PSE users, and from
supervisors of PSE personnel can provide important feedback about
the strengths and weaknesses of the PSE. Table 11 summarizes the
PSE evaluation techniques, the environments where they might best
be applied, and the advantages, disadvantages, and relative cost
of each technique.
5 Conclusion and Recommendations
A broad review of relevant literature shows that no single tool
can effectively measure and evaluate an activity as complex and
intangible as knowledge work, or a performance support environment
(PSE) for knowledge workers. This study concludes that a "toolkit"
of five evaluation techniques, each applicable to a specific workgroup
setting, may best assess the feasibility and usefulness of a PSE:
The first step in selecting the appropriate tools for evaluation
is to categorize the type of knowledge work performed within the
environment. The potential PSE site should be analyzed for four
composite attributes: (1) complexity, (2) volume per job, (3)
time per job, and (4) repetition.
A workgroup should first be categorized as either a knowledge
work-intensive (KW) professional environment, or a production
office. A professional office is one whose primary function requires
professional level workers, where the work is not substantially
repetitive, and where the clerical work is performed in support
of professional work. A production office is an office whose primary
function is the performance of a small number of repetitive, clerical-level
tasks, e.g., claims processing, order entry, and call centers.
Once the workgroup is categorized, an appropriate evaluation tool
should be selected. The Information Economics (IE) cost-benefit
analysis is recommended as the first step in deciding which PSE
impacts to evaluate. It is also recommended that a quality assessment
be done in conjunction with any other evaluation tool.
Cited
Anthony, G. Michael, "IE's Measure Work, Write Standards
for White Collar Workers at Financial Institution," Issues
in White Collar Productivity (Industrial Engineering and Management
Press, Institute of Industrial Engineers, 1984), pp 84-87.
Bernard, Paul, "Structured Project Methodology Provides Support
for Informed Business Decisions," IE (March 1986),
pp 52-57.
Beruvides, M.G., and D.J. Sumanth, "Knowledge Work: A Conceptual
Analysis and Structure," Productivity Management Frontiers-I
(Elsevier Science Publishers B.V., 1987), pp 127-138.
Brassard, Michael, The Memory Jogger-A Pocket Guide of Tools
for Continuous Improvement (Goal/ QPC, 1988).
Cleary, B.A., "Company Cares About Customers' Calls,"
Quality Progress, vol 26, No. 11 (November 1993), pp 69-73.
Cox, Thomas, "The Myth of the Commodity Database or How To
Pick the Best Technology for You," Oracle Integrator
(January/February 1993), pp 19-21.
Drucker, Peter F., Management (Harper & Row, 1974).
Helton, B. Ray, "Achieving White-Collar Whitewater Performance
by Organizational Alignment," National Productivity Review
(Spring 1991), pp 227-244.
Helton, B. Ray, "Quality and the Bottom Line, Part 2: The
Company's Side," The Quarterly Observer (March 1992),
p 5.
Helton, B. Ray, "Quality and the Bottom Line, Part 3: The
Inner View," The Quarterly Observer (April 1992),
p 3.
Helton, B. Ray, "More of the Right Stuff," The Quarterly
Observer (September 1993), p 5.
Liggettt, Trevino, and Lavelle, "Activity-Based Cost Management
Systems in Advanced Manufacturing Environments," in Parsaei
et al. (eds.), Economic and Financial Justification of Advanced
Manufacturing Technologies (Elsevier, Amsterdam, 1992).
National Research Council, Information Technology in the Service
Society (National Academy Press, Washington, DC, 1994).
Parker, Marilyn M., Robert J. Benson, and H.E. Trainor, Information
Economics Linking Business Performance to Information Technology
(Prentice-Hall, Englewood Cliffs, NJ, 1988).
Peterson, I., Probing a Computer Productivity Paradox," Science
News (1 Jan 1994), p 7.
Rittenhouse, Robert G. "Productivity and the Microcomputer,"
Management of Technology III (Institute of Industrial Engineers,
1992).
Roach, Stephen, "Services Under Siege-The Restructuring Imperative,"
Harvard Business Review (September-October 1991), pp 82-83.
Sassone, Peter G., "Cost Benefit Methodology for Office Systems,"
ACM Transactions on Office Information Systems,
vol 5, No. 3 (July 1987).
Sassone, Peter G.. "A Survey of Cost-Benefit Methodologies
for Information Systems," Project Appraisal, vol 3,
No. 2 (June 1988), pp 73-84.
Sassone, Peter G., "Office Productivity: The Impacts of Staffing,
Intellectual Specialization and Technology" (The Georgia
Institute of Technology-School of Economics, September 1991),
pp 1-35.
Sassone, Peter G., "Three Approaches for Estimating the Value
of Office Work," Office Technology and People, vol
6, No. 1 (1992).
Sassone, Peter G., "Survey Finds Low Office Productivity
Linked to Staffing Imbalances," National Productivity
Review (Spring 1992), pp 147-158.
Uncited
Johnson, H. Thomas, "It's Time to Stop Overselling Activity-Based
Concepts," Management Accounting (September 1992),
pp 26-35.
Kelly, Robert, and Janet Kaplan, "How Bell Labs Creates Star
Performers," Harvard Business Review (July-August
1993), pp 128-139.
Partovi, "An Analytic Hierarchy Approach to Activity Based
Costing," International Journal of Production Economics,
vol 22 (1991), pp 151-161.
Appendix A: Taxonomy of KWS Benefits and Capabilities
KWS Implementation Scoresheet
The following Scoresheet (Figure A1) is divided into four blocks
to facilitate the discussion of how to complete and utilize it.
Each block has a specific function that is detailed in Figures
A2 to A5.
BLOCK 1 The value codes appropriate to each evaluation class
are entered in the "value code" column.
The value codes are detailed in block 4. For
example, if improving performance is important to
you, enter a "4" in the code column next to
that class (Figure A2).
BLOCK 2 Each benefit category is related to each value
class by entering the appropriate relation code
in the corresponding columns. The relation codes
are detailed in block 4. For example, if you determine
that reduced rework is strongly related to Performance
Improvements, enter a "4" in the #2-value-column
of that category [B] (Figure A3). The "Ind
Benefit Scores" column on the right side of block 4, is
where the sum of the products for each category
is entered. Figure 3 shows an example where a "16"
is entered-a value of 4 times a relation of 4.
Figure 4 shows a more complete example.
BLOCK 3 The
sum of the individual scores from block 2 is entered in
the space to the right of "SCORE FOR KWS IMPLEMENTATION"
(Figure 5).
BLOCK 4 The codes used to rate value classes
and benefit categories are explained in this block.
These are not hard quanitiative values,
but are "fuzzy" values, i.e., your
interpretation of the values and benefits of these items to
the process being examined. The definitions given
are, therfore, purposely ambiguous so that they
can be applied to any situation.
Primary Objectives
The primary objectives of the Knowledge Worker System are:
Benefit I. Effectiveness Improvements
This benefit encompasses the capabilities described in benefits
II - V.
Benefit II. Rework Reductions
Capability: Process Model Documented
Capability: Institutional Knowledge Captured
Benefit III. Efficiency Improvements
Capability: Shared Task Schedule
Benefit IV. Focus Improvements
Capability: Referential Information Linked to Tasks
Benefit V. Work Elimination
Capability: Tasks Automated & Linked to Tasks
Appendix B: An Analysis of the Techniques and Application of
Information Economics to KWS
Introduction
Information economics (IE) is an effort to provide a more complete
means, when compared to simple return on investment (ROI), of
evaluating potential projects for a company. IE is a form of decisionmaking
and therefore, the evaluation technique is not limited to evaluating
projects, but can be used in many decisionmaking situations. The
application of this technique to evaluating potential implementation
sites for the Knowledge Worker System (KWS) would be very useful.
IE could provide an excellent approach of determining which performance
trends to track for that KWS pilot site. The following text is
an analysis of the techniques used in Information Economics and
its extension to KWS.
Analysis of IE
Background
A typical comparison of possible projects is based on accounting
type of information, i.e., projected costs, projected benefits,
and return on investment. The comparison may involve several projects
competing for resources or it may simply be a single project competing
with the current situation. The problem with all of this is that
it ignores a number of factors that may effect the success of
the project(s) simply because these factors do not lend themselves
to dollar quantification. IE provides a methodology for including
nonquantifiable factors in the analysis of projects. (For a detailed
presentation see Parker, Benson, Trainor 1988.)
It is important to note that IE attempts to make explicit some
of the biases implicitly applied in evaluating alternative projects.
For example, the organization's aversion to risk may be recognized
as a factor in rating projects. Different organizations may come
to different conclusions when evaluating the same project.
The most important factors in evaluating projects are recognized
and given relative weights. A firm very interested in ROI may
use a weight of 10 versus a weight of 2 for competitive response.
Then each project is ranked for each factor. In the case of ROI
and other quantitative measures, actual numbers can be applied.
The weights for each factor are applied to each project's rankings
for the factors and a total score is determined. A comparison
of these scores can give a good picture of the project(s) that
are best suited to this organization's vision.
The application of IE can be observed in a "tool" developed
by Oracle Corp. called CB-90 - Cost Benefit for the Nineties.
CB-90 breaks the analysis down into three factors: Tangible cost/benefit
analysis, Intangible cost/benefit analysis, and Intangible risk
analysis (Semich 1994). The latter two can be further subdivided
into business and technical groupings (Cox 1993). These two articles
with another (Pastore 1992) provide a valuable insight into the
application of the IE theory.
The IE Methodology
Information Economics focuses on value rather than the more limited
concept of benefit. It should be noted that much of the terminology
used in IE represents an application to the nongovernmental organization.
This does not decrease its applicability to governmental structures.
It simply means that some adjustments will be required. IE uses
six classes of value summarized as follows:
Each of these value classes is assigned a weight for the organization
in which the IE analysis is occurring. Eventually every proposed
project will be evaluated in each value class for its effect and
then a summarized score is calculated for each project.
IE applied to KWS
Background
Typically, with IE, an organization starts with a vision and proceeds
to establish the criteria and their weights to be used in deciding
which projects best fit into its vision. In applying IE to KWS
we must start with a set of potential benefits attributable to
KWS and then set up a methodology for evaluation in differing
environments. The following illustration clarifies this difference.
PERFECT Company places great value on never making a mistake on
a customer's order. Their vision is focused on this error free
goal. Their evaluation of any project will weight heavily its
affect on perfection. CHEAP Company on the other hand takes great
pride in being the cheapest price on the market and naturally
will weight any project heavily for its affect on price. The typical
IE evaluation would take place in the environment of the subject
company and would have criteria and weights established based
on the vision of that company. The approach we must make in evaluating
KWS is to develop a set of criteria in a tool such that the vision
of the implementation prospect can be accounted for.
In applying information economics methodology to KWS implementation
it is important to remember that IE is used to estimate the value
of a choice to an organization. If IE allows you to estimate this
value before the event, then it can also be applied to estimating
the value of the decision after the event. As the majority of
the work will have been accomplished in the initial estimate,
the work involved in the second estimate will be greatly reduced.
The specific means of evaluating the selected criteria is a separate
decision. The IE process establishes what is deemed important
by the user. The appropriate measures and tools can then be selected.
Consider the example mentioned above. The PERFECT Company's overriding
concern is perfection. Therefore, it will want to evaluate the
level of perfection achieved after the decision is made, that
is, if it were to use IE in the same manner suggested here.
Suppose that one of the benefits of KWS is that it makes the scheduling
more efficient, and that one member of the implementation team
has rated this as a very important benefit. Part of the determination
of the KWS productivity for this implementor would be the improvement
of scheduling. Other factors would also have to be included in
this evaluation of productivity, but one measure that must be
taken is the change in the scheduling function.
One of the important points of this technique is that the productivity
change is based on the factors important to the user. If the cost
of the change is of relatively low importance to the user, this
can be accounted for in this technique. Likewise, the technique
can account for the user for whom the costs are very important.
It could be argued that two users in similar circumstances might
then produce widely different productivity changes. The real measure
of the productivity change is what the user perceives and if the
user has established a vision then this perception should be the
best measure of utility.
It is important to note that this technique ties the measure of
productivity to the implementor's vision. This vision should reflect
a concern about the "customer," for as Peter Drucker
(1974) noted, the purpose of an enterprise is to create a customer.
An Outline of the Technique
To use IE techniques to evaluate a specific project-the implementation
of KWS-we must first establish the benefits of KWS. The following
section discusses a set of potential benefits.
Our list of benefits is something we take to each potential implementation
site. At each site we must then establish the values of the organization.
This second step allows us to plug these figures into a worksheet
and produce a rating for the implementation of KWS at that site.
As there is no alternative project(s) to compare KWS to simply
compare it to the status quo. Of course, if KWS is implemented,
we will later compare the post-implementation and the pre-implementation
scores to establish the productivity change. The score for the
status quo, however, is not necessarily zero, although it could
be. The status quo may have a very negative score (in those situations
where collapse is imminent). It could have a significantly positive
score-in those situations where operations are running well. The
comparison would indicate situations where the impact of KWS would
be significant and minor.
The evaluation tool then starts with a group of established benefits.
The second step is to determine the weights to be applied to the
six values of IE and to score the KWS benefits for the organization.
This process must be done by the organization itself.
The Benefits of KWS
The benefits of KWS can be organized in many different ways. One
approach is to separate the benefits into three broad categories:
Knowledge
How-to-do-it information will exist on the system. Formal rules
and manuals will be accessible along with learned-on-the-job information.
Both new and experienced workers will be able to perform complex
tasks with less references to external sources.
Files and data associated with particular tasks will be accessible
via KWS. This means that previous examples of a task will be easily
available for reference and that the processes currently being
utilized will be accessible. This reduces the time spent looking
for information and it increases the accuracy of the work performed.
Information associated with a task will be kept associated with
that task. This reduces the time spent looking for missing information.
Procedures will be easier to document. The procedures can grow
from the actual work being performed.
Control
Individuals will be able to see what their priorities are for
the day, week, and further into the future. They will therefore
be able to concentrate on the higher priority items.
Individuals will be able to see what they've done in the past.
Managers can optimize the abilities of the people available by
reassigning work and or priorities as situations change.
Annual leave, training, and other such activities can be scheduled
in less time and more effectively.
Individuals can perform as members of the group by accessing schedule
and other information and then acting on it.
Action
Repetitive tasks will be automated reducing the amount of time
necessary to prepare reports, write memos, gather data, etc.
All actions will be do-able from KWS. This reduces the time and
complexity of moving from software environment to software environment.
The KWS environment will reduce the time and effort involved in
accessing equipment, data and facilities.
An IE Toolset for KWS
Applying IE methodology to the implementation of KWS is not a
simple process -you can't simply pick one from column A, one from
column B, and so on. However, it doesn't need to be a difficult
process either. The use of a "Toolset" is what reduces
the difficulty of the process. We understand the workflow by using
a structured interviewing process. A layered survey gains the
value information we need and a spreadsheet calculates the productivity
value we are looking for. The first tool, the structured interviewing
process, has already been referenced. The second and third tools
will be discussed below.
We must elaborate on the values that the organization has and
apply them to the KWS application benefits. We would conduct a
survey, or rather a layered survey, where we first seek to establish
a consensus on the general vision of the organization. From there
we then proceed to the next layer and establish consensus on the
weights the organization would apply to the various value classes.
The third layer would entail a detailing of the value classes
and weighing each of them. From this information, we would be
able to transfer the weights to a spreadsheet that would be linked
to the KWS benefits. A fourth layer would be to estimate the potential
for each KWS benefit. Some benefits depend for their relevance
on the organization's environment. For example, if the reduction
in the access to external references is one of the KWS benefits,
but the subject organization never does this, there is no potential
to this benefit in this environment. Once this information has
been established and entered to the spreadsheet, it would automatically
produce a score for the implementation of KWS in the organization.
Important to this process is getting an estimate from the organization
of how well it feels it currently performs or satisfies each of
the values. This would produce the current rating to compare with
the KWS rating.
Summary
This approach to evaluating the benefits of KWS in a particular
environment has a very broad range of application. It can be done
very simply in a very short time and yet produce a good evaluation.
It can also be done in a more complete manner to produce more
detailed and better evaluations. As such, the amount of effort
invested in the evaluation need not determine the applicability
of the results. The quality of the effort is much more important.
This emphasis is quite different from many evaluation techniques
where the quantity of effort is the controlling factor.
The participation of the members of the area being studied is
crucial with the IE approach. Taken farther these individuals
could, with some assistance, conduct the studies themselves. Reaching
concensus among study participants is a significant by-product
of the IE process.
For further reading on Information Engineering:
Parker, Marilyn M., Robert J. Benson, and H.E. Trainor, Information
Economics: Linking Business Performance to Information Technology
(Prentice-Hall, Englewood Cliffs, NJ, 1988).
Drucker, Peter F., Management (Harper & Row, 1974).
Semich, William J., "Here's How To Quantify IT Investment
Benefits," Datamation, vol 40, No. 1 (7 January 1994),
pp 45-48.
Cox, Thomas, "The Myth of the Commodity Database or How To
Pick the Best Technology for You," Oracle Integrator
(January/February 1993), pp 19-21.
Pastore, Richard, "Many Happy Returns," CIO,
No. 5 (15 June 1992), pp 66.
Appendix C: Time Logging Introduction and Instructions for
Knowledge Workers
Introduction
A select group of HQUSACE managers and professionals has been
cooperating with USACERL in the initial development of the Knowledge
Worker System, a PC- and LAN-based computer application for supporting
and automating some of their administrative tasks. The members
of the initial KWS user group are key players in the PPBES cycle.
Because the development, implementation and possible future extensions
of KWS absorb scarce Army resources, it is important that KWS
be cost justified. This involves identifying and measuring the
various benefits and costs associated with the system. The best
approach to doing this is to begin with a baseline work analysis
(to document work patterns prior to implementing KWS), and then
to analyze work patterns after KWS has been implemented. A comparison
of these two work pattern snapshots will serve as the basis for
identifying and quantifying the impact of the system.
The economic benefits of knowledge work frequently are difficult
to quantify because most knowledge work is complex in content
and impact. However, a new approach for measuring the impact of
productivity interventions (such as KWS) on knowledge work has
been developed and used successfully in several dozen recent private
sector studies. This new methodology is called Work Profile Analysis.
It is based on the simple but powerful concept of the "intellectual
content" of work.
In general, organizations pay their employees based on the intellectual
content of the work that they are capable of performing. Engineers
with advanced training and much experience are paid more than
new and inexperienced engineers. An engineer with management training,
ability and experience is paid more than an engineer without those
qualifications. Similarly, program and budget analysts with more
training and experience are paid more than those with less. And
of course, analysts and engineers are paid more than secretaries
and clerks.
At the same time, workers do not spend 100 percent of their work
time doing the work that their training and experience qualifies
them to do. For example, managers, engineers and analysts may
spend only a fraction of their time in work that could, in principle,
be delegated to lesser skilled and lesser paid employees. Indeed,
research has shown that pattern to be the norm rather than the
exception. Of course, in many organizations, it is the shortage
of workers to whom work may be delegated that is responsible for
this pattern of work. For example, and quite simply, there often
are not enough secretaries and clerks in an office to handle all
the secretarial and clerical tasks, and therefore many of those
tasks must be done by professionals and managers.
By analyzing how workers at each position in an organizational
hierarchy typically spend their time (in terms of the intellectual
content of the work that they are doing), and by factoring into
the analysis the typical number of hours worked and the total
cost (wages or salary, overtime, fringe benefits) of those workers,
the actual (implicit) cost of different types of work (management
work, engineering work, analysis work, secretarial work, ...)
in that organization can be calculated.
As a simple example of this
methodol
ogy,
suppose
workers
in an
engineer
ing
organiza
tion
typicall
y spend
their
time as
shown
in
Table
C1Table
C1.
For
simplici
ty,
assume
that
everyone
works
40
hours
per
week
for 46
weeks
per
year
(this
assumes
6 weeks
total
of
vacation
,
holidays
, sick
days
and
training
days
per
employee
).
Finally,
suppose
that
the
average
annual
total
cost
(salary
or
wages
plus
fringe
benefits
) to
the
organiza
tion of
a
person
in each
position
is
given
in the
last
column
of the
table.
Example Work Profile Matrix
With these assumptions, the implicit cost to the organization
of management level work is $76.49 per hour, of professional engineering
is $51.33 per hour, and of administrative support work is $18.12
per hour. (Determining these values involves a mathematical economic
model. You can check the figures by noting that the values uniquely
and exactly account for the average salary in each position.)
To check these values for managers, note that managers spend 552
hours per year doing management level work (46 weeks x 40 hours/week
x 30%), 736 hours per year doing engineering work (46 weeks x
40 hours/week x 40%), 276 hours per year doing support work (46
weeks x 40 hours/week x 15%), and 276 hours per year in nonproductive
work (46 weeks x 40 hours/week x 15%). Then note that : (552 hours
of management x $76.49/hour) + (736 hours of engineering x $51.33/hour)
+ (276 hours of support work x $18.12/hour) + (276 hours of nonproductive
time x $0.00/hour) = $85,000. Similar checks can be made for the
engineers and secretaries in our example.
These implicit costs can be used to estimate the value of a computer
system, such as KWS. Continuing the simple illustrative example,
suppose that KWS could absorb 60 percent of the administrative
support work and 33.3 percent of the nonproductive work done by
engineers, and suppose that the saved time is redirected into
professional engineering work. The new row in the work profile
matrix would be:
Engineers 0% 80% 10% 10% $65,000
The value to the organization of this shift in the engineers'
work profile would be $18,889.44 per engineer per year (20% x
40 x 46 = 368 additional hours of engineering work per engineer
per year @ $51.33 per hour.) This calculation assumes that the
additional engineering time is productive. That is, it assumes
that there is a continuing "backlog" of engineering
work to do, and that this additional engineering work is as worthwhile
as the other engineering work.
In reality, of course, actual cases are far more complex than
the example described above. Nonetheless, these same ideas, incorporated
in more complex models, can be (and have been) successfully used
to evaluate virtually any productivity intervention in a knowledge
work environment.
This Study
The information needed for this study is "time log"
data, that is, data on how everyone spends their time. We will
use this data to develop a productivity snapshot of several USACE
Headquarters offices. The data collection and analysis is based
on two critical assumptions:
To complete the study, we need your cooperation in completing
time logs. We ask that you make your entries on your time log
sheet once each hour for your assigned three consecutive work
days.
So that we get an accurate picture of the work in your organization,
it is important that you conscientiously complete the time logs.
After reading these instructions and completing your first entries
will take only a minute or two.
Categories of Work in the USACE Headquarters
There are six major broad types of work in the organization, along
with several important finer distinctions. The six major work
categories are:
Each of these categories of tasks is explained in detail in "Appendix
D: Dictionary of Work Categories and Example Tasks." In addition
to these six major categories of tasks, there are several other
activities which normally occupy the time of HQUSACE personnel.
These are: professional development, nonproductive time, community
service, travel, and personal time.
Finally, for these few days, completing the time log is an activity
which will take a few minutes of each hour. These activities are
explained in the same document. Tasks usually are composed of
distinct sub-tasks. For example, the task of preparing an engineering
analysis might include the following sub-tasks:
This task would be considered an engineering task because one
or more of the sub-tasks must be performed by an engineer. However,
some of the sub-tasks could be characterized as "support"
or "set-up" work; and some of the sub-tasks could be
characterized as "principal" or "core" work.
For example, sub-tasks a, c, d, e, i, j, and k are partly or completely
"support" level work. That is, these are sub-tasks that
could be delegated to a competent and properly trained administrative
assistant, if one were available. On the other hand, items b,
f, g, and h are "principal" level sub-tasks. They involve
professional engineering, and they cannot be delegated to a support
worker.
In general, the "support" or "set-up" sub-tasks
of a professional or managerial task involve activities like searching,
identifying, accessing, looking up, downloading, locating, transmitting
or communicating documents, data or information; or they involve
routine office skills like typing, filing, data entry, simple
database queries, scheduling, copying, faxing, etc.
As you complete your time log, you will be asked to distinguish
between time spent in "principal" level work and time
spent in "support" level work.
Engineering and Program/Budget Analysis work also can be classified
according to the minimum level of education, training and experience
necessary to accomplish that work. In the HQUSACE, we can distinguish
two levels of engineering and two levels of program/budget analysis
work. These are:
This is work that can be done by a competent entry level engineer
or analyst. This is someone with the educational background appropriate
to the profession, who has had the appropriate training but who
has less than several years experience. Junior engineering or
analyst level work would include much of the routine engineering
or analysis work performed in the HQUSACE. As a general guide,
junior level engineering work would be professional engineering
work that could be delegated to a GS-12 or lower engineer. Junior
level program/budget analysis work would be analysis work that
could be delegated to a GS 11 or lower analyst.
This is either engineering or analysis work of sufficient complexity,
uniqueness, impact or importance that it should be handled by
engineers or analysts with more than several years experience;
or it is planning, directing and overseeing the engineering/analysis
work of others. As a general guide, senior level engineering work
would be professional engineering work that should be performed
by GS 13 or higher engineers. Senior level program/budget analysis
work would be analysis work that should be done by GS 12 or higher
analysts.
Instructions
Please look at a time log sheet. The first column on the left
lists the work categories discussed above. Listed first are the
six major types of work (with engineering and program/budget analysis
each separated into junior and senior level work), then the remaining
activities which are sometimes a significant part of nearly everyone's
day, then a catch-all "Other" category, and finally
a row labeled TOTAL MINUTES.
The rest of the columns on the sheet are for your entries for
each daily time period. The first and last of the columns on the
sheet are for your entries for each daily time period. The first
and last of these time periods are open ended (BEFORE 7:00 AM
and AFTER 6:00 PM), and the others are exactly 1 hour long.
Before using the time log sheet to record your daily activities,
you should review the "HQUSACE Dictionary of Work Categories
And Example Tasks" (Appendix D). This will help you understand
how each category is being defined for the purpose of this study.
It will also ensure consistency in everyone's interpretation of
the work categories.
Make your entries in the time log sheet at the end of each hour
(or as close to that time as possible). Enter the number of minutes
of each type of work for that time period in the appropriate space
on the time log sheet.
The work categories listed in the time log sheet are intended
to be complete and nonoverlapping. Therefore, the entries in each
column should add up to 60 minutes. The only exception might be
your first and last time periods of each day. For example, if
you start work at 8:40 AM or leave work at 4:30 PM, then your
first and last periods will contain only 20 minutes and 30 minutes,
respectively. Please be sure to check your entries so that you
have accounted for all your time in each column. In particular,
you should account for all 60 minutes in each hour that is not
the first or last hour of the work day.
In most cases, assigning your actual work in the time log categories
should be straightforward. However, because work in HQUSACE is
varied and complex, not every work activity can be explicitly
anticipated by the time log. Therefore, in those cases, you should
first review the document, HQUSACE DICTIONARY OF WORK CATEGORIES
AND EXAMPLE TASKS, and then use your best judgement to assign
your time to the closest category. Note that the listed tasks
under each main type of work are intended only as examples. If
the work that you did reasonably fits that category, you should
log your time under that category even if your specific tasks
are not explicitly listed. However, if none of the categories
truly fit the work that you did, then use the "OTHER"
category and add a brief explanation on the back of the sheet.
As a guide in assigning your time to the proper category, use
the delegation test. Ask yourself: What is the lowest level to
which this work could reasonably be delegated? If the work could
be done by a secretary or clerk, then that work is administrative
support work. If the work could not be done by a typical secretary
or clerk, but it could be done by someone with some technical
training (e.g., a PC specialist), then the work is technical support
work. If the work is engineering work but it could be delegated
to a junior engineer, then the work is junior level engineering
work. And so on.
In posing and answering this delegation question, don't worry
about current staffing limitations (that is, for example, a secretary
could do this s but none are available). Simply assume that there
is adequate staffing to handle any delegated work.
If you need help, or if you have any questions about filling out
your time log sheet, you should call Beverly Thomas at 217/373-7284.
Please remember to make your entries on the form at the end of
each hour, or as close to that time as possible!
Critical Information
Please do not make up data if you forget to log a certain
period. Rather, we will extend your assigned 3-day logging interval
so that a total of 3 days work is ultimately captured.
Please do not rely on your memory to fill out the form only
once or twice a day, instead of once each hour. Extensive
experience clearly shows that hourly logs, while an admitted inconvenience,
are the best way to produce accurate results.
Do not be concerned if the days which you log do not appear
to be "typical" for you. Do not adjust your data
to make it look typical! We fully expect that no one's 3 days
will be typical for him or her. However, by combining the results
from all employees, the computed averages will be statistically
reliable.
Do not be reluctant to log nonproductive time. This category
helps us understand and quantify your need for additional support.
Remember, our assumption is that you are getting your job done
the best way possible, even if that entails spending a lot of
time on nonproductive activities.
Your time log sheets will be treated confidentially. The
time log sheets will be analyzed by an external consultant. Only
summarized data (no names) will be reported.
Appendix D: Dictionary of Work Categories and Example Tasks
Project Administration
These are projectrelated tasks (research or reimbursable) that
must be performed by the project manager. These can be referred
to as "plan, program, schedule and budget" project.
Some examples of this work include:
Project Execution
These are tasks that directly support a research or reimbursable
project. These can be referred to as "execute" research
or reimbursable project. Some examples of this work include:
Technology Transfer
These are tasks that lead to the dissemination of information
about research or reimbursable project. Some examples of this
work include:
Technical Support Tasks
These are support tasks which demand some technical skill or knowledge.
Some of these tasks are done by nearly everyone. The key question
to ask is, "Is this work related to my project?" These
tasks should not be directly related to your reimbursable or research
project.
Some examples include:
Administrative Support Tasks
These are nontechnical tasks that can be performed by someone
with general office skills and some onthejob training. Some of
this work is done by nearly everyone. Some examples include:
Serving on or Supporting Government Committees
This refers to all the time that you spend working on government
committees that are not projectrelated. This work might be done
alone or in meetings with others. Some examples of these tasks
include:
Professional Development
This refers to all the time you spend gaining, maintaining or
updating the knowledge and skills necessary to do your job. This
includes all formal and informal training and education. This
work might be done alone or in group sessions. Some of this work
is done by nearly everyone. Some examples include:
Nonproductive Time
This refers to work time during which no work gets done. Despite
everyone's best efforts, nonproductive time is an almost inevitable
part of each day. Some typical nonproductive activities include:
Community Service Activities
This refers to all the time that you spend during working hours
engaged in community support activities as a representative of
the government. This work might be done alone or in meetings with
others. Some examples of these tasks include:
Travel Time
This refers to time that you spend traveling on company business
during normal working hours, during which time you are not getting
any work done. If, for example, you are getting work done while
you are riding in a car or airplane, then you should log that
time under the appropriate work category. Do not include intrabuilding
transit time in this category (log that time in the nonproductive
category), but do include interbuilding transit time (e.g., going
from Building #1 to Building #3).
Personal Time
This refers to the nonwork periods during the work day. Examples
include:
Completing the Time Log
This refers to the time you spend reading these instructions and
filling out time log sheets. If you spend time referring back
to these instructions or documents, getting help in completing
the time log, distributing or collecting these forms or reviewing
them for completeness, or helping someone else with their form,
log that time under Completing Time Log, as well.
Meetings
Use this area to indicate any meetings that do not fall under
other categories. For example, SAEDA security briefings, Division
meetings, information meetings about some workrelated (but nonproject)
related topic such as the new USACERL phone system, etc. Many
of your meetings that are not project related will be logged either
here or under "Professional Development."
Other
Please note-on a piece of paper or separate file-any activities
that you have performed during the past hour that do not fit under
any other category described above. Record the # of minutes spent
in this undefined category in the "Other Type of Work"
section.
(Eccles, in Helton 1992)
Work Range
Specialist * Professional Support Clerical
Non-repetitive P P S
Repetitive S P P
Non-routine P P S
Routine P P
Individual P P
Group S E S
Sequence-Dependent E P
Work Structure
Shifting Objectives P P S
Fixed Objectives S P P
Control
Discretionary P P S S
Non-discretionary P P
Cognitive Effort
Very Substantial P S
Substantial P
Limited E P
*P = primary; S =
secondary; E =
either
Complexity Time per Job Repetitions Volume
per
Worker
Component Level -> H M L H M L H M L H M L
Knowledge Work X X X X X X X X
Production Office X X X X X X X X
Proceduralized Work X X X X X X X X 
Figure1. Work categorized by four work components.
Figure 2. Information economics.
Figure 3. Scoresheet for KWS implementation.
Position
Managerial Professional Support Non-Productive
Managerial 30% 20% 20% 30%
Professional 5% 35% 35% 25%
Support 0 0 75% 25%
Total 5% 15% 55% 25%
Position
Managerial Professional Support Non-Productive
Managerial 60% 20% 10% 10%
Professional 15% 55% 15% 15%
Support 0 0 80% 20%
Total* 10% 30% 45% 15%
Cost Drivers
Products Assembly Inspection Material Handling
Gadgets 4 7 4 X 250/50 = 20
Widgets 6 2 6 X 40/50 = 4.8
Activities
Total Hrs/yr Cost/hr Cost/yr
Assembly 2 x 40 x 50 = 4000 $10.00 $40,000
Inspection 2 x 40 x 50 x 70% =2800 $10.00 $28,000
Material Handling 2 x 40 x 50 x 30% =1200 $10.00 $12,000
Activities
Cost/yr No. of Drivers Cost Per
Driver
Assembly $40,000 1.3 million components $.0308/component
Inspection $28,000 1.0 million tests $.0280/test
Material handling $12,000 2.72 million feet $.0044/foot
Assembly Inspection Material Handling ABC Unit Cost
Product No. Cost No. Cost No. Cost $0.4073
Gadget 4 $0.0308 7 $.0280 20 $0.0044 $0.2618
Widget 6 $0.0308 2 $.0280 4.8 $0.0044 $0.2618
Product Total Direct Labor Costs % of Direct Costs Overhead Allocation Total Production Costs Conventional Unit Costs % Error in
Unit Costs
Gadgets $ 12,300 30.75% $ 12,300 $ 24,600 $.2460 -39.6%
Widgets $ 27,700 69.25% $ 27,700 $ 55,400 $.3693 +41.1%
Totals $ 40,000 100% $ 40,000 $ 80,000 Element
Example
Driver Weekly requirement to report division's staff hours (by
project) for labor accounting system
Task Prepare Labor & Time Sheet for 60 knowledge workers
Activities
Time spent on task (per week) Activity 1: 0.10 hour
Activity 2: 0.40 hour
Activity 3: 1.10 hour
Activity 4: 0.30 hour
Activity 5: 0.60 hour
Total: 2.50 hours per week
Implicit cost of task (per year) $ 12.95 * 130 = $ 1,683.50 labor cost X total No. of hours =
annual cost
Figure 4. Tools for quality assessment.
Technique Recommended Environment Pro Con Cost*
Work Profile Analysis All Complements PSE purpose; Can be automated Data intensive; Time consuming Low-med
Direct vs Indirect Production Office KW** Intuitive; Identifies tangential activities Data intensive; Time consuming Low-med
TSTS All Easy to implement; Intuitive; Attractive to management May measure the
wrong attribute Low
ABC Production Office KW** Identifies costs Data intensive;
Time consuming Med-high
Workflow Production Office Systematic; Good analysis tool Time consuming Med-high
Quality All By definition, measure what's important Can be time consuming Low-med
* Costs can
be reduced if
initial
information
is collected
as part of
process
building
performed by
the PSE
implementation
team.
** Also
appropriate
in knowledge
work
environments
where
processes are
stable and
structured.
Note that
many DOD
organizations
have
performed ABC
as part of
other process
modeling
initiatives.
Figure A1. Scoresheet for KWS implementation.
Figure A2. First block of KWS implementation scorecard.
Figure A3. Sample block containing "Indicate Benefit Scores" column.
Figure A4. Sample benefit score entry.
Figure A5. Sample completed benefit scores.
Management Work Professional Engineering Work Administrative Support Work Nonproductive Work Average Annual Cost
Managers 30% 40% 15% 15% $85,000
Engineers 0% 60% 25% 15% $65,000
Secretaries 0% 0% 90% 10% $30,000
Back to Table of Contents