For a reminder of the criteria for full credit, please read below:Reading response submissions are due by close of business (COB, 5:00pm) Monday, the day before the lecture.Your submissions should be about 1 page, single-spaced, 11pt font.Full credit will be given for submitting on time with reasonable content, including both summary question/comment, and meeting length requirementsThe response should contain a summary of key points of reading, in your own words.Please also include either or both an insightful question about some aspect of the material, or a comment on how the reading relates to other course concepts or examples from your own experiences.e
zhang_2003.pdf
Unformatted Attachment Preview
Journal of Biomedical Informatics 36 (2003) 23–30
www.elsevier.com/locate/yjbin
Using usability heuristics to evaluate patient safety of medical devices
Jiajie Zhang,a,* Todd R. Johnson,a Vimla L. Patel,b Danielle L. Paige,c and Tate Kuboseb
a
School of Health Information Sciences, University of Texas Health Science Center at Houston, 7000 Fannin, Suite 600, Houston, TX 77030, USA
b
Department of Biomedical Informatics, Columbia University, Vanderbilt Clinic, 5th Floor 622 West 168th Street, New York, NY 10032, USA
c
Department of Psychology, Rice University, Texas, USA
Received 1 July 2003
Abstract
Objective. To modify the traditional heuristic evaluation method of assessing software usability so that it can be applied to
medical devices and used to evaluate the patient safety of those devices through the identification and assessment of usability
problems.
Design. Heuristic evaluation, a usability inspection method commonly used for software usability evaluation, was modified and
extended for medical devices. The modified method was used to evaluate and compare the patient safety of two 1-channel volumetric
infusion pumps.
Results. The modified heuristic evaluation method was successfully applied to medical devices. One hundred and ninety-two
heuristic violations were categorized for 89 usability problems identified for Pump 1, and 121 heuristic violations were categorized
for the 52 usability problems identified for Pump 2. Pump 1 had more usability problems with high severity ratings than Pump 2.
In general, Pump 1 was found to have more usability issues that are likely to induce more medical errors.
Conclusions. Heuristic evaluation, when modified for medical devices, is a useful, efficient, and low cost method for evaluating
patient safety features of medical devices through the identification of usability problems and their severities.
2003 Elsevier Inc. All rights reserved.
Keywords: Medical error; Patient safety; Heuristic evaluation; Usability; Medical device; Infusion pump
1. Introduction
The medical error report from the Institute of Medicine [1] has greatly increased peopleÕs awareness of the
frequency, magnitude, complexity, and seriousness of
medical errors. As the eighth leading cause of death in
the US, ahead of motor vehicle accidents, breast cancer,
or AIDS, medical errors occur in many medical situations. One such situation is the use of medical devices.
Medical device use errors are a common source of patient injury and death. In many cases, medical devices
have user interfaces that are so poorly designed and
difficult to use that they invite a variety of human errors.
FDA data collected between 1985 and 1989 demon*
Corresponding author. Fax: +713-500-3929.
E-mail addresses: [email protected] (J. Zhang), todd.r.
[email protected] (T.R. Johnson), [email protected] (V.L.
Patel).
1532-0464/$ – see front matter 2003 Elsevier Inc. All rights reserved.
doi:10.1016/S1532-0464(03)00060-1
strated that 45–50% of all device recalls stemmed from
poor product design (including problems with software)
[2,3]. Furthermore, the FDA recognizes that a poorly
designed user interface can induce errors and operating
inefficiencies even when operated by a well-trained,
competent user. In response, the FDA has revised its
Good Manufacturing Practice regulations to include
specific requirements for product usability [2]. They
have also published guidelines for interface design and
usability testing [3] and produced a continuing education article that specifically covers usability issues [4].
Other research suggests that injuries resulting from
medical device use errors far exceeds injuries arising
from device failures [5].
In this paper, we modify a usability engineering
technique called heuristic evaluation for the evaluation
of usability problems in medical devices. Through the
identification of usability problems, we can indirectly
identify medical devicesÕ potential trouble spots that are
24
J. Zhang et al. / Journal of Biomedical Informatics 36 (2003) 23–30
likely to cause medical errors. In Section 2 we describe
the usability heuristics modified for the evaluation of
medical devices, the scale for severity rating of usability
problems, and the procedure of carrying out a heuristic
evaluation. In Section 3 we select two 1-channel volumetric infusion pumps and perform a heuristic evaluation on them. In Section 4 we discuss the implications of
heuristic evaluation in the evaluation of patient safety in
medical device use.
2. Background
Numerous research reports, medical error reports,
and other documents show a clear link between usability
problems and user error [6,7]. The FDAÕs report, Do it
By Design [3], describes a variety of errors resulting
from medical device interface design. For example, a
physician treating an infant with oxygen set the flow
knob between 1 and 2 L/min and then later noticed that
the infant was not receiving any oxygen. Even though
the knob rotated smoothly, the device was designed to
deliver oxygen only when the knob was set on a number,
not between numbers. Adding detents to the knob, so
that it would click onto a number, and providing visible
feedback of the rate of flow could have greatly decreased
the chance of this type of error. In another example,
found in the FDAÕs manufacturer and user facility device experience database (MAUDE), a nurse tried to
program an infusion pump to deliver 130.1 ml/h of a
drug, but inadvertently programmed the pump to deliver 1301 ml/h, because the decimal point on the pump
was designed to operate for numbers no greater than
99.9. When the nurse pressed ‘‘1 3 0 . 1’’ the device ignored the decimal point key-press. Since simply ignoring
the decimal point clearly results in a number that is 10
times larger than intended, this error could be prevented
by designing the device to alert the user whenever the
decimal point is pressed after more than two digits have
been entered. The alert could inform the user of the
problem and then force the user to reenter the number.
Human factors engineering is a discipline that seeks
to design devices, software, and systems to meet the
needs, capabilities, and limitations of the users, rather
than expecting the users to adapt to the design. A
complete human factors engineering analysis for medical devices or software systems includes four major
components: user, functional, task, and representational
analyses [8]. User analysis is the process of identifying
the characteristics of existing and potential users, such
as their expertise and skills, knowledge base, educational
background, cognitive capacities and limitations, perceptual variations, age related skills, cultural background, personality, time available for learning and
training, frequency of system use, and so on. User
analysis can help us design systems that have the right
knowledge and information structure that match that of
the users. Functional analysis is the process of identifying critical top-level domain structures and goals that
are largely independent of implementations. It is more
abstract than task and representational analysis because
it does not involve details of task processes and representations. Task analysis is the process of identifying
system functions that have to be performed, procedures
and actions to be carried out to achieve task goals, information to be processed, input and output formats
that are required, constraints that must be considered,
communication needs that have to be satisfied, and the
organization and structure as well as the information
categories and information flow of the task. One important function of task analysis is to ensure that only
the necessary and sufficient task features that match
usersÕ capacities and are required by the task will be
included in system implementations. Task analysis can
be conducted at different levels of detail. A keystrokelevel model lists the sequence of keystrokes and other
physical actions required to complete a specific type of
task [9,10]. For instance, the task may be programming
an infusion pump to deliver 500 ml at 100 ml/h and the
keystroke-level model lists the keys or buttons the user
must press to complete this task. A cognitive task
analysis includes cognitive operations, such as the goal
of entering the rate (which may be accomplished physically in different ways, such as by using up–down arrows or typing in the rate), or determining the volume to
be infused based on the physicianÕs order and the drug
concentration. Representational analysis is the process
of identifying an appropriate information display format for a given task performed by a specific type of user
such that the interaction between the users and the
system is as direct and transparent as possible. With
direct interaction interfaces, users can directly, completely, and efficiently engage in the primary tasks they
intend to perform, not the housekeeping interface tasks
that are barriers between users and systems. The file
browser in Microsoft Windows uses a direct interaction
interface to move, delete, and rename files, whereas
command line systems (e.g., MS DOS) do not.
These four types of analyses, when combined and
applied to a single product, can reveal the full range of
usability issues, which are essential for an understanding
of patient safety implications of the product. Heuristic
evaluation, the method we modified and used in the
current study, is primarily at the level of representational
analysis and is only one of the major techniques at this
level. We focus on heuristic evaluation in this paper,
because it has been shown to be one of the most costeffective methods of finding usability problems. We discuss the details of heuristic evaluation in the next section,
along with a description of the techniqueÕs advantages,
limitations, areas of application, and alternative techniques that my be used to augment heuristic evaluation.
J. Zhang et al. / Journal of Biomedical Informatics 36 (2003) 23–30
3. Heuristic evaluation
Heuristic evaluation is an easy to use, easy to learn,
discount usability evaluation technique used to identify
major usability problems of a product in a timely
manner with reasonable cost [11–14]. This technique
requires three or more evaluators to independently apply a set of usability heuristics to a product, identify
violations of the heuristics, and assess the severity of
each violation.
Heuristic evaluation is a type of usability inspection
method, which refers to a class of techniques in which
evaluators examine an interface for usability issues. Inspection methods are considered an informal usability
evaluation method, because they rely on heuristics and
the experience and knowledge of the evaluators. In
contrast, empirical techniques assess usability by testing
an interface with real users, and formal techniques, such
as task analysis, use models and formula to measure
usability [12]. Formal methods are often difficult to use,
so the most common usability evaluations are inspection
and empirical methods.
During a heuristic evaluation, experts walk through
the interface and identify elements that violate usability
heuristics. This method has become extremely popular
in the realm of usability evaluation due to its low cost,
low time commitment, and ease of application [13].
Evaluators can conduct the evaluation in a few hours
with minimal training. This method has been traditionally used to evaluate websites as well as desktop
software applications, and it is typically used to point
out software interface difficulties to be addressed in the
design process. It can be applied to paper or electronic
mock-ups or prototypes as well as completely implemented designs.
In this paper, we modify the heuristic evaluation
method to address three issues in the evaluation of
medical devices. First, we use it to discover usability
problems that are likely to cause medical errors. Second,
the ‘‘discount’’ nature of heuristic evaluation may also
prove useful for the comparison of patient safety features of alternative medical devices, as is the case of the
purchasing process of medical devices. Third, heuristic
evaluation may also be a good tool for medical device
manufacturers to improve the patient safety features of
their products during the design and redesign processes.
3.1. Fourteen usability heuristics
Nielsen [13], as the major researcher who developed
the technique of heuristic evaluation, described 10 major
heuristics that should be followed by good user interface
design. Shneiderman [15] also described eight golden
rules that all good user interface designs should follow.
Based on the ten heuristics by Nielsen, the eight golden
rules by Shneiderman, and our own considerations, we
25
state the following 14 heuristics with semantic tags
(words in the brackets), names, general descriptions, and
specific information about the heuristics. We call these
14 heuristics the Nielsen–Shneiderman Heuristics because these heuristics are mostly based on their work.
1. [Consistency] Consistency and standards. Users
should not have to wonder whether different words,
situations, or actions mean the same thing. Standards and conventions in product design should be
followed.
a. Sequences of actions (skill acquisition).
b. Color (categorization).
c. Layout and position (spatial consistency).
d. Font, capitalization (levels of organization).
e. Terminology (delete, del, remove, rm) and language (words, phrases).
f. Standards (e.g., blue underlined text for unvisited hyperlinks).
2. [Visibility] Visibility of system state. Users should be
informed about what is going on with the system
through appropriate feedback and display of information.
a. What is the current state of the system?
b. What can be done at current state?
c. Where can users go?
d. What change is made after an action?
3. [Match] Match between system and world. The image
of the system perceived by users should match the
model the users have about the system.
a. User model matches system image.
b. Actions provided by the system should match
actions performed by users.
c. Objects on the system should match objects of
the task.
4. [Minimalist] Minimalist. Any extraneous information is a distraction and a slow-down.
a. Less is more.
b. Simple is not equivalent to abstract and general.
c. Simple is efficient.
d. Progressive levels of detail.
5. [Memory] Minimize memory load. Users should not
be required to memorize a lot of information to carry out tasks. Memory load reduces usersÕ capacity to
carry out the main tasks.
a. Recognition vs. recall (e.g., menu vs. commands).
b. Externalize information through visualization.
c. Perceptual procedures.
d. Hierarchical structure.
e. Default values.
f. Concrete examples (DD/MM/YY, e.g., 10/20
/99).
g. Generic rules and actions (e.g., drag objects).
6. [Feedback] Informative feedback. Users should be given prompt and informative feedback about their
actions.
26
7.
8.
9.
10.
11.
J. Zhang et al. / Journal of Biomedical Informatics 36 (2003) 23–30
a. Information that can be directly perceived, interpreted, and evaluated.
b. Levels of feedback (novice and expert).
c. Concrete and specific, not abstract and general.
d. Response time.
• 0.1 s for instantaneously reacting;
• 1.0 s for uninterrupted flow of thought;
• 10 s for the limit of attention.
[Flexibility] Flexibility and efficiency. Users always
learn and users are always different. Give users the
flexibility of creating customization and shortcuts
to accelerate their performance.
a. Shortcuts for experienced users.
b. Shortcuts or macros for frequently used operations.
c. Skill acquisition through chunking.
d. Examples:
• Abbreviations, function keys, hot keys,
command keys, macros, aliases, templates,
type-ahead, bookmarks, hot links, history,
default values, etc.
[Message] Good error messages. The messages should
be informative enough such that users can understand the nature of errors, learn from errors, and recover from errors.
a. Phrased in clear language, avoid obscure codes.
Example of obscure code: ‘‘system crashed, error code 147.’’
b. Precise, not vague or general. Example of general comment: ‘‘Cannot open document.’’
c. Constructive.
d. Polite. Examples of impolite message: ‘‘illegal
user action,’’ ‘‘job aborted,’’ ‘‘system was
crashed,’’ ‘‘fatal error,’’ etc.
[Error] Prevent errors. It is always better to design interfaces that prevent errors from happening in the
first place.
a. Interfaces that make errors impossible.
b. Avoid modes (e.g., vi, text wrap). Or use informative feedback, e.g., different sounds.
c. Execution error vs. evaluation error.
d. Various types of slips and mistakes.
[Closure] Clear closure. Every task has a beginning
and an end. Users should be clearly notified about
the completion of a task.
a. Clear beginning, middle, and end.
b. Complete 7-stages of actions.
c. Clear feedback to indicate goals are achieved and
current stacks of goals can be released. Examples
of good closures include many dialogues.
[Undo] Reversible actions. Users should be allowed to
recover from errors. Reversible actions also encourage exploratory learning.
a. At different levels: a single action, a subtask, or
a complete task.
b. Multiple steps.
c. Encourage exploratory learning.
d. Prevent serious errors.
12. [Language] Use users’ language. The language should
be always presented in a form understandable by the
intended users.
a. Use standard meanings of words.
b. Specialized language for specialized group.
c. User defined aliases.
d. UsersÕ perspective. Example: ‘‘we have bought
four tickets for you’’ (bad) vs. ‘‘you bought four
tickets’’ (good).
13. [Control] Users in control. Do not give users that impression that they are controlled by the systems.
a. Users are initiators of actors, not responders to
actions.
b. Avoid surprising actions, unexpected outcomes,
tedious sequences of actions, etc.
14. [Document] Help and documentation. Always provide
help when needed.
a. Context-sensitive help.
b. Four types of help.
• task-oriented;
• alphabetically ordered;
• semantically organized;
• search.
c. Help embedded in contents.
3.2. Severity rating scale
The heuristics are used to check the interface of the
device design. If a heuristic is violated, it is given a severity rating based on the following scales [12]:
0, not a usability problem at all;
1, cosmetic problem only. Need not be fixed unless
extra time is available;
2, minor usability problem. Fixing this should be given low priority;
3, major usability problem. Important to fix. Should
be given high priority;
4, usability catastrophe. Imperative to fix this before
product can be released.
As a guideline for rating the problems, we consider
the proportion of users who will experience it, the impact it will have on their experience with the product,
and whether the usability problem will be a problem
only the first time they encounter it, or whether it will
persistently bother them. A persistent problem with a
major impact that most users will encounter will get the
highest severity rating.
3.3. Procedure
Typically 3–5 usability experts independently evaluate
the user interface of a product and each of them generates
a separate list of heuristic violations according to the
14 heuristics described above. We find it convenient for
J. Zhang et al. / Journal of Biomedical Informatics 36 (2003) 23–30
each evaluator to begin with a tabular form, shown in
Table 1, where each row contains where the problem occurs (place of occurrence), a description of the problem,
the heuristics violated, and the severity rating (though the
ratings are filled in at a later stage). It is best to agree ahead
of time on what to call each place of occurrence. For instance, ‘‘physical interface’’ ma …
Purchase answer to see full
attachment
