How eye tracking gives players a new experience in video games

The Division video games uses eye tracking technology to help target the enemy. Tobii EyeX/YouTube screengrab

Tracking people’s eye movements is a concept that for a long time has captured people’s imagination. More often than not, the technology has been depicted as part of rather dystopian futures: in the movie Minority Report, police could track your every move through your eyes; in Fifteen Million Merits, part of the television series Black Mirror, adverts would pause when you looked away, forcing you to pay attention.

The reality of eye tracking, however, can be a lot more fun than that. In particular, our research group has been interested in the applications of eye tracking for gaming.

From the lab to the home

Eye trackers used to cost tens of thousands of dollars and were precious pieces of equipment at psychology labs. While expensive high-speed trackers still have their place in eye movement research, eye trackers for gaming purposes can now be bought for just a few hundred dollars.

Examples include the Tobii EyeX, the Steelseries Sentry and the EyeTribe. We are also starting to see laptops
that ship with built-in eye trackers.

But these devices still lack a killer app, a compelling use case that provides a compelling reason for people to adopt the technology.

Gaming might very well suit this role.

An eye tracker is made of little more than a camera and a few LEDs to be attached to the bottom of a screen. The difference to a regular webcam is that they operate in the infra-red spectrum so that the variation in people’s eye and skin colour does not interfere in the tracking.

Sophisticated algorithms then convert the image from the camera into a pair of coordinates that represent where you are looking at on the screen: the gaze point. Though deceptively simple, these coordinates offer a wealth of opportunities for developers to incorporate into their games.

What to do with the gaze point?

The simplest mechanic that can be built for the eyes is to use them as a cursor, similarly to how we use the mouse.

Variations of this include using them to select and place chess pieces, to move the weapon cross-hair in a first-person shooter game, and to replace the mouse enabling disabled users to play.

Gaze data can also be combined with the keyboard and mouse to create more immersive experiences. In Assassin’s Creed Rogue, when the player looks at a point in the environment the character faces that direction, allowing for a more natural exploration of the environment.

Tom Clancy’s The Division incorporates several gaze mechanics, including aiming, throwing grenades, looking for cover, and many more.

The advantage of using gaze is that developers can still reach players who do not own an eye tracker yet, but offer an extra experience for those who do.

The future of eye tracking in games

One of the coolest aspects of gaze as an input method is that it goes well beyond issuing explicit commands to games.

First, the eyes play an important role in social interactions between people and this can be incorporated into games. For example, characters can get upset if you look away when they are speaking or maybe engage in a conversation when you first notice them.

Second, the eyes reveal a lot about our cognitive processes, which can be used by the game’s artificial intelligence to adapt the gameplay. For example, researchers have explored how to use players’ gaze to infer their strategies in Hex, to predict their actions in Super Mario Bros and to modify the narrative in interactive storytelling. This gives games an almost mind reading ability that can wow players (or possibly freak them out).

Third, eye trackers can provide professional and hardcore players with a powerful analytics tool for them to evaluate and improve their performance. For example, players can record their eye movements throughout the game session and later watch the video to check whether they were paying attention to the right areas at the right times. A heatmap of gaze points can also give players a snapshot of their overall visual attention and highlight areas that require more focus.

A game-changer or just a gimmick?

Eye tracking has been around for a long time, but it is yet to reach the wide consumer market.

Recent examples, such as Pizza Hut’s gaze-enabled menu, suggest that there certainly are interesting applications for the technology, but they still have to prove their value beyond gimmicky demos.

The Nintendo Wii Controller and the Microsoft Kinect have shown that gaming can be a powerful platform to demonstrate the capabilities of unusual input devices.

We will keep our eyes open to see if the same will happen for gaze.

Eduardo Velloso receives funding from the Victorian State Government and Microsoft through their contributions to the Microsoft Research Centre for Social Natural User Interfaces (SocialNUI).