Surface computing is about integrating the physical world and the virtual world through the use of vision-based touch. While Microsoft's Surface product is the best-known implementation of surface computing, it is far from the only one. Expanding university research on touch continues to make use of vision-based touch as a foundation, which in turn will help move surface computing toward full commercialization.

by Geoff Walker and Mark Fihn

THE TERM "surface computing" (sometimes called "tabletop computing") describes a specialized computer graphical user inter-face (GUI) in which (1) the keyboard and mouse are completely replaced by a touch-sensitive display and (2) users interact with common and intuitive objects rather than conventional GUI elements such as windows, icons, and drop-down menus. The goal of surface computing is to integrate the physical world and the virtual (digital) world more closely so that digital information becomes immediately and easily available when users interact with a physical object or an environment.

Conceptual Examples

One example of surface computing is a horizontal touch display that has been used on a trial basis in retail mobile-phone stores. The user/prospect places two physical phones on the display's surface. The software driving the display identifies the phones and immediately displays a comparison of the two phones' features, specifications, and pricing. The user can then interact with the information using his hands to explore details or modify the way the phones are compared. Another example involves placing a digital camera on the display surface and having the photos in the camera automatically copied to the display, where the user can interact with them using multi-touch finger-gestures such as flicks, pinches, rotations, etc. The photos can be transferred to a mobile phone simply by placing the phone on the display surface and dragging the photos over to it.

History

Around the mid-1990s, a variety of companies and institutions began conducting research on surface computing, including Alias|Wavefront, Microsoft, MIT Media Lab, Mitsubishi Electric Research Labs (MERL), New York University, Sony Computer Science Labs, the University of Toronto, and Xerox PARC.^1,2 MERL's announcement of the DiamondTouch interactive table in 2001 heralded one of the first commercially available surface-computing products, but it was viewed as a research curiosity and was not fully commercialized. Microsoft's hype-filled announcement of the Surface product in May 2007 caught the public's attention in a big way. The widespread publicity that Microsoft's announcement received caused the Surface product to become synonymous with surface computing. However, innovations related to surface computing are not unique to Microsoft, and there are numerous other efforts enabling the technology.

Technology

As with all touch-screen technologies, there are two interrelated components – the display device and the touch-sensing device. On the display side, surface computing can work with any type of display, including flat panel, rear projection, and front projection. On the touch side, the choices are more limited. While some early implementations such as DiamondTable used capacitive sensing, essentially all current implementations of surface computing use infrared (IR) vision-based sensing; this requires one or more IR imaging cameras to be positioned so that an image of the entire screen can be captured. This means that today all surface computers use either rear or front projection, which eliminates true "tabletop" use – unless the surface computer is itself a table, like Microsoft Surface (illustrated in Fig. 1).

IR Light Source

There are currently three methods of supplying the IR light that is received by the vision-based camera in surface computing. These methods, Diffused Illumination (DI), Frustrated Total Internal Reflection (FTIR), and Diffused Surface Illumination (DSI), are explained in the following paragraphs.

• DI: Diffused Illumination can be used with either front- or rear-illumination systems. Rear DI (used in Microsoft Surface) utilizes infrared light projected on the screen from below the touch surface. A diffuser is placed on the top or the bottom of the touch surface. When an object touches the surface, it reflects more light than the diffuser (or objects in the background), and the extra light is sensed by a camera. Depending on the diffuser, this method can also detect hover above the screen and can identify objects placed on the surface. In the case of front DI, infrared light is projected on the screen from above the touch surface, such that a shadow is created when an object touches the diffused surface and can then be similarly recognized by a camera.

• FTIR: Popularized in the touch-screen world by Jeff Han when he was at NYU (he's currently the Founder and CEO at Perceptive Pixel), the concept of Frustrated Total Internal Reflection is a physical condition related to differences in the refractive indexes of adjacent materials. When light passes from one material to another with a higher refractive index at an angle of incidence greater than the specific angle (described by Snell's Law), then no refraction occurs in the material, and light is reflected. This method traps infrared light in an acrylic overlay, which is frustrated (scattered) at the point of a touch; the scattered light is then recognized by camera-based imaging. Figure 2 illustrates the concept.

• DSI: Diffused Surface Illumination uses a special acrylic to distribute the IR evenly across the surface. This method relies on small particles inside the acrylic, which function like tiny mirrors. When IR light is injected into the edges of the acrylic (as in FTIR), the particles redirect the light to the surface and spread it evenly. When a user touches the surface, the light is scattered and seen by the vision-based camera as a blob of IR light.

Significance

Vision-based touch systems have not yet achieved any substantial commercial penetration. The technology is in a state somewhat similar to that of projected capacitive ("pro-cap") in the early-to-mid 2000s – it's a niche technology waiting for a breakthrough. In the case of pro-cap, the breakthrough was Apple's decision to use it in the iPhone; that decision had an immense effect not only on the technology, but also on the entire touch industry.

The authors believe that vision-based touch has even more potential to change the world than pro-cap. The latter is, after all, simply a substitute for a mouse and keyboard in interacting with the standard GUIs running on the iPhone OS and Windows. Interaction with computers has not been changed in any fundamental way by pro-cap touch screens; it's just been made simpler and more fun, especially on small mobile devices. In contrast, surface computing at its core is an attempt to totally change the way people interact with computers. Putting a digital camera down on an interactive surface, having the photos it contains spill out onto the surface, interacting with the digital photos on screen through multi-touch gestures, and sharing the photos with several other people in a tabletop environment is very different than tapping icons and selecting menu items on an iPhone.

There is one other characteristic of vision-based touch systems that's significant, and that is the fact that they can be assembled from inexpensive standard parts and open-source software.⁴ No other multi-touch-capable touch technology can be obtained in this way. As a result, the majority of touch research being conducted in university research labs uses vision-based touch as its foundation. This means that there is an expanding body of work being developed on vision-based touch along with steadily increasing knowledge and understanding of the technology; this increases the probability of a breakthrough that will drive the technology toward full commercialization.

Fig. 1: Schematic format of Microsoft's Surface computer product. The components numbered in blue are as follows: (1) acrylic tabletop touch surface with a diffuser; (2) 850-nm infrared light source directed at the underside of the touch surface; (3) infrared camera (one of four with a combined total resolution of 1280 x 960 pixels); (4) Texas Instruments' DLP projector running at 1024 x 768 pixels; (5) desktop computer running a customized version of Microsoft Vista. Source: Microsoft.

Hardware and software developers remain excited about viable commercial uses of surface computing and continue to innovate. A few data points on progress in this area include the following:

• MERL's DiamondTouch interactive table, mentioned earlier in this article, has been licensed exclusively by startup Circle Twelve, Inc., which is marketing it as a collaboration tool. The DiamondTouch table enables multiple simultaneous inputs such that each user can be separately identified. The technology uses front projection and capacitive sensing.

• SMART Technologies offers a vision-based direct-touch technology, which is commercially available and widely used in education and collaboration applications. SMART offers touch tables and rear-projection interactive whiteboards that incorporate the technology.

• GestureTek markets a multi-touch table in sizes from 30 to 55 in.; installed applications include wayfinding and entertainment. GestureTek holds the world's record for the largest surface-computing table at 6 m long (located at the Eureka Tower in Melbourne, Australia).

• Microsoft has expanded the concept of surface computing to go beyond just tabletop applications, demonstrating ideas related to spherical surfaces (both on the exterior, as with a globe) and on the interior (as with a dome).

• Microsoft has demonstrated surface computing using photo-sensors located behind thin-form-factor LCDs (ThinSight⁵). By using an electronically switchable diffuser, Microsoft has also demonstrated that images can be recognized and displayed well beyond the surface of the screen (SecondLight⁶). The recognition of physical objects, either as an interface device that can identify objects, or as a projection device that inserts data on surfaces beyond the surface of the screen, is a computationally formidable user-interface task.

• Several recent projects have investigated the possibility of linking together geographically separated surface computers in order to create a shared workspace for remote collaboration, as if participants are co-located around the same tabletop.

Conclusion

In the almost 3 years since Microsoft announced Surface, there has been an accelerating flow of ideas and information about surface computing – just try Googling "touch table"! The technology holds the promise of changing the way people interact with computers, going well beyond (for example) applying touch in the replacement of conventional user interfaces in appliances. There is an expanding body of work on multi-touch, object recognition, direct manipulation, 2-D and 3-D gestures, and related fields that continue to enable innovation in the area of surface computing. The technology is ripe for a breakthrough.

References

¹http://www.billbuxton.com/multitouchOverview.html

²http://spectrum.ieee.org/computing/hardware/ slideshow-tabletop-computers/0

³J. Y. Han, "Low-cost multi-touch sensing through frustrated total internal reflection," Symposium of User Interface Software and Technology (2005).

⁴http://www.maximumpc.com/article/features/maximum_pc_builds_a_multitouch_surface_computer

⁵S. Izadi, et al., "ThinSight: integrated optical multi-touch sensing through thin form-factor displays," ACM International Conference Proceeding Series (2007).

⁶S. Izadi, et al., "Going beyond the display: a surface technology with an electronically switchable diffuser," Symposium on User Interface Software and Technology, ACM (2008). •

Fig. 2: This figure illustrates how FTIR can be used to sense touch. A rear-projection screen (diffuser) is attached with a small air gapto the underside of a sheet of acrylic. Infrared LEDs inject light into the polished edge of the acrylic; TIR causes the light to remain trapped within the sheet. A baffle blocks light with a higher angle of incidence near the edge of the acrylic. When a finger touches the surface of the acrylic, it "frustrates" TIR and causes light to scatter out through the acrylic towards a vision-based camera equipped with an IR bandpass filter. Source: Media Research Laboratory, New York University.³