Minding Your Business

A telephone call to Roel Vertegaal’s lab may cause a pair of plastic-foam eyeballs to wiggle. Those peepers are attached to a desktop gadget that Vertegaal says could presage a generation of what you might call digital secretaries–particularly insightful ones at that. If Vertegaal looks at the shaking eyeballs, they’ll suddenly stop and stare back at him and then patch the call through. If instead,

Vertegaal doesn’t establish eye contact with the little pop-eyed gizmo on his desk, an answering machine kicks into gear. That’s because his digital secretary could tell in a glance that Vertegaal wasn’t interested in taking the call.

Such interactions between Vertegaal, director of the human media lab at Queen’s University in Kingston, Ontario, and the gadget, known as eyePROXY, are an experiment in a new style of human-machine interactions. Known as attentive-user interfaces, these combos of gadgetry and software are designed to make our growing staff of machines accommodate human behaviors. The goal is to render that entourage of technology more helpful and less annoying.

Cell phones, pagers, personal digital assistants (PDAs), laptops, and car navigation systems–the list of these devices lengthens as electronic intelligence and communications links infiltrate household and office items. There are now even prototype cooking spoons that tell you what temperature the batter is. As this corps of gizmos swells, a barrage of blinking lights, ring tones, beeps, vibrations, and other cues calls out for attention. “The volume of notifications . . . is becoming so large that people are having trouble dealing with it,” Vertegaal says.

Consider the demands placed, for example, on nurses, with their many patients and even more health-care gadgets to mind. Add their family, community, and social obligations, and they’re often at the edge of information overload, he says.

To alleviate this type of problem without having people limit their access to information, Vertegaal and other computer scientists are designing new software and hardware. “We are on a mission to change the way it feels to work with computers,” says Eric Horvitz of Microsoft Research in Redmond, Wash. The fruit of all these efforts will be that digital devices “become a lot less like tools and a lot more like companions and collaborators who understand your intentions and goals,” Horvitz predicts.

The scientists want to create the equivalent of the ideal personal human assistant, a flesh-and-blood helper who would recognize what input you need or want at any moment.

One way that researchers are pushing toward that goal is to develop attentive systems, such as eyePROXY, that monitor a person’s eyes. Other systems scan a person for position, motion, gestures, and other body language. Yet others tune in to electronic sources, including schedules and sensors indicating where in the world the person is and what he or she is doing. With such information, the system then can ask itself questions about its user: Does he want to take an incoming phone call? Is she too wrapped up in a videoconference to bother?

Eye’ll be watching you

Despite the current enthusiasm for multitasking and the proliferation of gadgets that demand a chunk of our attention, most people can effectively handle only a few inputs at a time. Psychologists find that an average person can hold only about seven unrelated chunks of information in his or her mind at any one moment.

Some computer scientists and engineers now see a chance to infuse psychological savvy into interactive devices. “We can take the results from psychology . . . and leverage them” by incorporating them into information tools that people use, Horvitz says.

New technologies, like eyePROXY, may provide machines with the power to observe, interpret, and communicate with their users on human terms. Some investigators say that there’s no better place to start than with the face.

“A huge amount of information can be gleaned from watching someone’s eyes,” says Daniel Russell of IBM’s Almaden Research Center in San Jose, Calif. For example, people often vie for one another’s attention by means of a complex interplay of glances that make and break eye contact.

In the late 1990s, the IBM group that Russell now leads developed a compact video camera that determines where a person’s eyes are pointing at any moment. The system exploits the red-eye effect that wrecks many a snapshot. In this context, light reflecting from the retina renders conspicuous the pupil’s location, which a machine can then use to track the person’s gaze.

Russell’s lab has incorporated the camera into a prototype computer interface that he says could make the Web more effective as an educational tool. The system observes exactly what sentence or image on a Web page a person is focused on and how long he or she may be spending on a particular item–say, a division problem. With those data, the interface might then direct the student toward help with long division rather than with multiplication.

Such systems may enable computers to become better tutors for educational courses disseminated over the Web, Russell says. Anticipating only a small market for the camera, IBM made its design available for free to other researchers interested in eye-gaze tracking.

Vertegaal’s group took up IBM’s offer. The researchers have increased the camera’s resolution and miniaturized its electronics so that the whole unit is now small enough to be mounted on a computer, a hat, or even a pair of eyeglasses.

The team has built the same technology into eyePROXY and communications devices, including cell phones, that recognize when a person is engaged in a face-to-face conversation. That way, even if his or her cell phone is on, it can take a message instead of permitting the phone to ring. This application might be considered a rudeness blocker.

Vertegaal’s group has also incorporated the camera into gadgets that it calls “eyePLIANCEs.” One of them, a floor lamp, turns on or off by voice command only when a person is looking at it. Another is a television that recognizes whether someone is watching it and, if not, pauses the action on its display. Such attentive devices can sum into an environment that can respond to or even anticipate a person’s needs, Vertegaal’s group proposes in the March Communications of the ACM.

A simpler sensor that can pick up information about eye behavior has been developed at the Massachusetts Institute of Technology (MIT). A small diode produces infrared light that’s reflected off the eyeball and detected by a photodiode. The device was created by a team led by Ted Selker, director of MIT’s Context-Aware Computing lab and former director of the IBM lab that created the eye-gaze tracker.

Named eye aRe units–a takeoff on the abbreviation for infrared–the devices can monitor such eye characteristics as the rate at which a person blinks.

That can provide a window onto a person’s mental and emotional state, Selker says.

For instance, a rate near the high end of the typical range of 1 to 6 blinks per minute may indicate that the person is experiencing stress or fear.

Selker has several ideas about how such a sensing ability might be used. In one test, he and his colleagues programmed an eye-monitoring digital assistant to use blink-rate to infer whether someone preferred one type of music to another. In another experiment, they monitored people having conversations and found indications of which ones got along well.

Mindful machines

Besides eye-monitoring technologies, interface designers are also experimenting with methods for gauging other aspects of a person’s state of being. Those include video systems that determine body position and head orientation and audio interfaces that listen to and interpret speech and other sounds in the environment.

For instance, for mobile individuals, Microsoft Research has made a prototype PDA souped-up with several sensors to automatically detect how the user intends to use it. These sensors include an accelerometer that tells the device whether it’s upright, a touch sensor that indicates whether the device is being held, and a proximity sensor that reports whether there’s a solid body–for example–a head, within arm’s length of the device. With the accelerometer, this prototype can also determine whether its user is walking.

If wirelessly networked to the person’s office gadgets, a PDA with even this limited bit of awareness about the person’s activities could help redirect messages and other information to that guy or gal on the go.

Even if gadgets become better at discerning the information, computing, or communications needs of their users, they–like able assistants–still need to know when and how to get the user’s attention.

For example, Selker and Ernesto Arroyo, also of MIT, compared whether brightening a light or heating up a mouse pad is the better way to grab a person’s attention. They found that the light was good at quickly producing awareness. The heated pad, although it took longer to attract interest, was better at holding that attention, says Selker. Actually, he adds, the heat might be especially effective for getting a visually overloaded person’s attention, say, when an emergency arises. The researchers reported these results in Miami last January at the International Conference on Intelligent User Interfaces.

Selker, Russell, and their respective teams have also been developing automotive versions of attentive interfaces. The systems are on the alert for signs of drowsiness, for example, and may blare the car radio, make the steering wheel quiver, or provide other feedback to jar the driver back into a more alert or safety-conscious state.

All in the family

Some skeptics, such as artificial intelligence pioneer John McCarthy, emeritus professor at Stanford University, ask whether making devices more attentive could make them more annoying. “I feel that [an attentive interface] would end up training me” to accommodate it rather than providing a worthwhile service, McCarthy says.

Rather than adding people-reading senses to machines, Microsoft Research’s Horvitz and his colleagues favor developing sophisticated software tools to analyze the continuous stream of personal information that’s already flowing through the computers and other digital equipment that people routinely use. This software sifts out signs of what the user is doing or might want to do in the near future.

The Microsoft researchers, for example, are developing ways to analyze and disseminate data to create systems that hold back low-priority distractions, such as an e-mail notice for discount Viagra, while–in a sensitive, courteous manner–pushing through urgent messages, such as the arrival of a crucial document in the mail room.

The team recognizes that examples abound of smart gadgets or software functions that aren’t smart enough. An overzealous digital assistant, such as the infamous animated paperclip that pops up in the word-processing program Microsoft Word, can annoy and get in the way.

Horvitz and his team recently developed a system, represented to computer users by a cartoon genie, that scans incoming e-mail, recognizes whether the message might require scheduling of an appointment, and offers to set one up. By slightly increasing the time delay before the genie sprang forward to volunteer its services, the interface transformed from something “terrible” to a seemingly “smart, intuitive person,” Horvitz recalls.

These researchers are also developing software systems that mine such information troves as e-mails, calendar appointments, logs of network use, and organizational charts identifying someone’s bosses and underlings. If unobtrusive information from passive observations by cameras and microphones is available, that’s included, as well.

The forte of the Microsoft Research group is developing mathematical models that can incorporate such information and–even without full knowledge about a person’s present location and activities–calculate probabilities of what he or she will want to do or might want to know. To do this, the researchers are trying to make the model assess the information in the less-than-precise way that people do.

More specifically, these models work by assigning the equivalent of dollar values to pieces of information to be delivered to someone and to the cost of interrupting that person to present the information. By these calculations, the software may opt to interrupt a person if he is gazing out the window but not if he is briefing the CEO.

“The magic of the kind of work we do here is that we do not change the way people work,” Horvitz says. “We don’t say, Hold your eyes in a certain way, but we pick up on all the things going on in your life.”

Even if attentive devices work seamlessly, all these possibilities for scrutiny may smack of Big Brother to some people. “That can be a potentially sensitive issue,” acknowledges Horvitz.

****************

If you have a comment on this article that you would like considered for publication in Science News, send it to editors@sciencenews.org. Please include your name and location.

To subscribe to Science News (print), go to https://www.kable.com/pub/scnw/
subServices.asp.

To sign up for the free weekly e-LETTER from Science News, go to https://www.sciencenews.org/subscribe_form.asp.