Forget Creepy Intel: SHORE Unlocks Your Face At A Glance, And It's Already In Use

If you thought Intel's plans for a viewer-watching Web TV box were intrusive, you might want to bury your face in your hands (and leave it there permanently) after seeing Fraunhofer's clever and creepy SHORE facial ID system. On show at CeBIT, SHORE can not only identify a face in a still image or real-time video stream, but figure out gender, age, and even what mood the person is in: happy, surprised, angry, or sad. Meanwhile, while Intel's home entertainment tracking system is already mired in controversy, Fraunhofer tells us commercial implementations of SHORE are already out in the wild.

In Fraunhofer's demo, a computer running SHORE was able to identify and classify multiple people walking in and out of frame, with the results of the analysis floated over each person on a wall display. The measurements happen almost instantaneously – the research institute says SHORE can identify a face at 107.5fps if it's directly facing the camera, while full analysis including facial expression detection is at 45.5fps – and the system can handle head tilts of +/- 60-degrees and head rotation of +/- 90-degrees.

So far so good, but it's the measurements not the identification which is what makes SHORE so impressive. Fraunhofer claims a 91.5-percent accuracy rate on face detection and a 94.3-percent accuracy rate on gender detection: by identifying the face, the eyes, nose, and mouth, and the rest of the facial shape, it can decide how happy or sad, angry or surprised they are. The extent of those criteria are displayed on red bars: when we smiled, it accurately picked up on that, while widening our eyes boosted our "surprised" rating.

More patchy was the age detection, which gives an estimate with a degree of confidence (so, for instance, SHORE could decide you're 38, with a range of +/- 8-years). That proved susceptible to being confused by the ambient lighting: with strong ceiling lights, for instance, those wearing glasses were often confused for someone much older, because the shadows of the glasses frames on their cheeks were mistaken for evidence of old age.

Nonetheless, it's a mighty impressive system in all, not least because of the incredibly low minimum specifications. Fraunhofer says SHORE will run on a single core of an Intel Core 2 Duo 6420 processor, under Windows XP, and with facial detection from anything down to an 8 x 8 pixel image (though you won't get the more complex analysis). It'll also run on mobile devices, such as smartphones, and can either be a standalone system or integrated into another, more complex monitoring package.

That flexibility – and the fact that Fraunhofer is licensing out the technology together with the offer of customizing it depending on client need – means the possibilities for implementation are far greater than, say, Intel's proposed advertising tailoring on the Web TV box. Market research is an obvious one, for instance a camera above a store window display to track reactions of those glancing in, as well as customizing advertising playlists depending on the demographics of those watching. Car dashboards could monitor drivers and ensure they were alert and calm, as well as better track which person was giving which spoken command.

In hospitals, the degree of pain to which patients are suffering could be monitored autonomously, helping the more efficient use of painkillers (and avoiding unnecessary suffering). Augmented reality games are another possibility, but Fraunhofer is also keen on the idea of using the SHORE technology to enhance "virtual actors" and "intelligent agents" for customer services and entertainment, reacting to those they are talking to, behaving appropriately for their mood, and even mimicking that mood themselves. In fact, Fraunhofer had a robotic head which, using a camera in the forehead, could replicate the viewer's expressions with animated eyes, mouth, and other elements.

Behind the scenes, the magic is in the huge amount of education Fraunhofer has given the system, teaching it to recognize common schema of mood and reaction from thousands of images of expressions. Called the Facial Action Coding System (FACS) it allows the computer to quickly calculate what each viewer is showing in a matter of milliseconds. That's even if there are dozens of people in the frame, too; Fraunhofer showed the camera a print out covered in face thumbnails – over a hundred of them, packed tightly together – and SHORE spotted them all and ran its mood analysis. The system has short-term memory, too; Fraunhofer tells us that faces aren't stored long-term, but there is a shorter-term caching system which can spot if a face was in-frame very recently, and collate all the data from each sighting. Each gets a temporary ID code, and a timer to show how long they were attentive for.

Perhaps most alarming is the fact that this isn't a simple piece of prescient, Minority Report-style research: SHORE is already out in the wild. Fraunhofer couldn't tell us all of its clients, but did confirm that market research firm GFK is using SHORE for its consumer surveys. There, participants simply allow the standard webcam on their computer to feed their expressions back to the server, as they watch a series of commercials or other content. Meanwhile, there are SHORE installations already watching passers-by from within store display windows, though Fraunhofer wouldn't be pressed on which retailers exactly are using it.

Meanwhile, you can try it for yourself. Fraunhofer offers a free trial version of SHORE to download, as a proof of concept, which you can find here. Those particularly paranoid might prefer to spend their time knitting balaclavas, however, as the possibility that you're being watched, analyzed, and generally figured out by a machine running something like SHORE is growing every day.