Rethinking the human factor
- By Paula Shaki Trimble
- Aug 28, 2000
A computer system that can be operated using a glove along with eye contact
and speech recognition may not be much of a fashion statement. But Rutgers
University researchers believe that it's more natural for people to communicate
electronically using such a system.
Rutgers has completed a three-year, $780,000 research project funded
by the National Science Foundation to develop systems that will replace
the keyboard and mouse with devices that track the user's gaze, respond
to voice commands and enable the user to move icons on the screen with a
gloved hand that registers motion.
During the project, the Army National Guard of New Jersey demonstrated
the ability of military planners to more accurately and more easily collaborate
on military planning procedures using the system, called the Speech, Text,
Image and Multimedia Advanced Technology Effort (Stimulate).
Stimulate is a joint initiative of NSF's Computer and Information Science
and Engineering Directorate, the National Security Agency's Office of Research
and Signals Intelligence Technology, the CIA's Office of Research and Development,
and the Defense Advanced Research Projects Agency's Information Technology
Rutgers researchers used a combination of prototype technologies to
study how people interact with computers employing systems that offer multiple
modes of communicating sight, sound and touch, said Edward Devinney, senior
associate director of Rutgers' Center for Advanced Information Processing
(www.caip.rutgers.edu), where the Stimulate research was conducted. The
research team plans to deliver its final report to NSF within a few weeks.
"We've found it is faster to use multimodal input," Devinney said. If
someone who's never used a computer is told to grab an object using a glove,
the tendency is to just grab in a natural way, he said, while it takes some
time to become comfortable with a mouse.
The system uses special software called Fusion Agent to teach the computer
to interpret what the user wants and to prioritize the activities, he said.
For instance, in a situation where a user looks at one thing but says another,
the computer may be programmed to always follow the voice command.
A key part of the system is a force-feedback glove, patented by Rutgers,
which reads gestures by detecting fingertip positions relative to the palm.
When a user points to or picks up an object on the screen, the user receives
feedback in the form of pressure from the glove. The glove weighs less than
The system also employs a gaze tracker. Rather than a headpiece, the
device is a unit mounted on the desktop that rotates to detect where the
user is looking. The user can direct a cursor just by looking at the computer
screen. Voice recognition and voice synthesizing software can understand
simple commands and respond audibly.
For instance, if a military user tells the computer to create a camp,
it will place the new camp where the user's eyes are pointed or where the
user points with the glove and then respond that it has created the camp.
"We've got quite a playground here," Devinney said.
Users can work together from different locations through a standard
Internet connection or other type of network. The users would launch Rutgers'
object-oriented groupware called Distributed System for Collaborative Information
Processing and Learning (Disciple), which integrates the inputs from the
various devices on the user end into logical commands.
Devinney said potential applications include battlefield management,
tele-medicine and other types of collaborative decision-making during which
users are mobile. The Rutgers team has suggested to the Defense Advanced
Research Projects Agency that it employ an enhanced version of the system
for designing the command post of the future. According to that proposal,
users would collaborate around a three-dimensional situational map, moving
assets around or identifying units and using the glove to retrieve information
about the unit's readiness and size.
Seven participants from the Training Technology Battle Lab at Fort Dix,
N.J., spent several months field-testing disaster-relief scenarios in which
military planners could collaborate remotely to decide how to deploy resources
and assess the terrain on a map-based display.
"The advantage is the ability to be able to see the map and the activity
on the map," said Brig. Gen. William Marshall, deputy State Area Command
commander for the New Jersey Army National Guard.
Marshall said the Rutgers Multimodal Input Manager led to the concept
of the digital battlefield, in which planners can test a possible movement
and see its impact.
Because moving an icon that represents a military unit automatically
generates a wide variety of technical information about the action, such
as position or number of personnel in the unit, the Rutgers system saves
considerable time and redundancy, Marshall said.
"If you move a flag on a flat map, you have to record the data in two
or three places. It's one entry vs. three with grease pencils," he said.
Devinney said the system has potential for helping people with disabilities
access computers. For example, the glove is being used elsewhere at Rutgers
as a physical therapy tool for people with hand injuries.
But he said brand-new developments are not ideal for such computer applications.
"The best chance of getting technology into the hands of the handicapped
is if you adapt something made in large quantities," he said.