Cell phones enable people to communicate almost regardless of where they are located. Since the main phone communication method is via voice, however, those who are deaf or hard of hearing need to use some alternate form of communication since vocal communication would be problematic. In Sweden and Japan, for example, there is enough bandwidth and cellular computing power to transmit full video of individuals speaking sign language to one another. In the US, bandwidth constraints and low phone power is not enough to allow for an intelligible conversation. New work by a team of engineers at the University of Washington has found a way to overcome this problem; the result is the MobileASL project, which allows people to communicate using American Sign Language (ASL) over cell phones.
While deaf individuals can communicate via non-verbal text messaging, SMS suffers from the lack of ability to convey other nonverbal cues that ASL offers. "Video is much better than text-messaging because it's faster and it's better at conveying emotion," said Jessica DeWitt, a UW undergraduate in psychology and a collaborator on the MobileASL project who is deaf.
The team at Washington, led by Prof. Eve Riskin, first tried to simply use low-resolution video that could be transmitted over the slow US networks. The quickly found that the video quality was no where near good enough to carry on a meaningful ASL conversation.
ASL uses more than just the motion of the hands: facial expressions convey important grammatical information, and the hands can point to various locations in three-dimensional space around the speaker—something difficult to capture on a two-dimensional video regardless of transmission speed. When thinking about carrying on an ASL conversation, one must also keep in mind the differences in the fluency of the speakers. A native ASL speaker will be able to fill in the blanks in choppy video in a way similar to how people fill in the holes in a poor cellular voice connection. Taking this all into account led the researchers to find a middle ground, capturing the details at a high enough quality for a meaningful conversation, while not exceeding the bandwidth readily available.
Since the hands and face are the most important parts of the body when communicating using ASL, the researchers came up with a methodology that allowed a high-quality encoding of the hands and face to be transmitted along with a low-quality encoding of the remainder of the body. The process used multiple steps to encode the video in real time using the x264 encoding standard. The software first identifies the important regions of the video using a skin detection algorithm, then by computing motion vectors determines how small the various macroblock levels must be to capture the areas of high motion in detail.
While some areas of the United States have access to the higher-speed 3G networks, not all do, so the researchers feel that the ability for everyone to communicate on 2G networks should be available. "The faster networks are not available everywhere," said doctoral student Anna Cavender. "They also cost more. We don't think it's fair for someone who's deaf to have to pay more for his or her cell phone than someone who's hearing."
The technology is working on a lab scale, but is not available to the public yet. A 20-person field trial is starting to get underway in Seattle, and the team is currently looking for volunteers.
Further reading (viewing):
- YouTube video explaining the MobileASL project
- Press release from the University of Washington
- Found via The Chronicle of Higher Education