Welcome to the Light Field
by Nikhil Balram
From the earliest days of history, humans have attempted to capture and display images. Cave paintings dating back 35,000 or more years show views of the world as seen by the ancient artist. The concept of photography as the capture of light using a recording medium is much younger, going back only to 1839, but the underlying concept of the camera obscura dates back to the sixth century. Paintings and photographs acted as static displays, showing various types of noteworthy images of the world. It was only 130 or so years ago that the idea of displaying the world in the form of moving pictures first took hold with the coming of films in the 1890s. The first commercial electronic displays, based on cathode-ray tubes (CRTs), only appeared as recently as 1922 and eventually enabled the concept of consumer television.
What is striking about this long history of capture and display is that almost all of it is based on two-dimensional imagery. The idea of enabling depth using stereoscopy was introduced by Charles Wheatstone in 1838. Since then, the notion of creating a perception of depth by presenting shifted images to the left and right eye has manifested itself many times – in the form of many generations of static stereoscopic viewers, printed images with lenticular layers creating distinct left and right views, 3D cinema using different color filters to separate left and right images, and, in the last decade, 3D cinemas and televisions using active or passive polarized glasses to enable separate shifted images for the two eyes. However, in each case the attempts at making three-dimensional viewing the norm failed to gain mass adoption — stereoscopic 3D has remained a niche. There may be many reasons for this failure but at least one prominent one is that presenting shifted left and right images on a single 2D image plane does not sufficiently approximate the way the human visual system sees the real three-dimensional world.
The continuous evolution of technology is once again presenting the dream of capturing and seeing three dimensions, this time using the light field, just like the human visual system does. But what is the light field, how is it captured and displayed, how does it take us from “flat land” to a true three-dimensional representation of the world, and when can we immerse ourselves in it? This special issue attempts to answer these questions through the two articles that follow.
The light field can be defined succinctly as the radiance that emanates from every point in the world and is visible to the human eye. The first article, “Light-Field Imaging and Display Systems,” is a summary of the 4-hour Sunday Short Course I taught at Display Week 2016 in May. Written by my colleague Dr. Tošić and me, the article provides a definition of the light field and explains various imaging approaches for capturing it and display architectures for showing it. As we explain in the article, light-field imaging is reasonably well understood as the capture of a four-dimensional data set with two spatial and two angular dimensions with a well-defined methodology for designing systems tailored to specific applications such as factory automation, medical imaging, and cinematic virtual reality.
On the display side, the situation is much less clear. Given the propensity of the marketing side of the consumer-electronics industry to be creative in its use of terminology, one can expect “Light-Field TVs” coming to a holiday season in the not-too-distant future. From a display-technologist and vision-scientist point of view, a light-field display is one that presents a light field to the viewer and enables natural and comfortable viewing of three-dimensional scenes. As explained in the article, for comfortable viewing a display needs to create a natural accommodation response for each eye that matches the convergence response in the human visual system, where these two are tightly coupled. There are two fundamental ways of doing this – either (1) by creating parallax across each eye that produces the appropriate retinal blur corresponding to the 3D location of the object being viewed (presenting multiple views per eye) or (2) by placing the object on the appropriate focal plane corresponding to its 3D location (providing multiple focal planes per eye).
Implementing either of these approaches, or a hybrid of them, requires tradeoffs that need to be made very carefully with the target application in mind. The most fundamental application choice is whether a display system is designed for group (multi-user) or personal (single-user) viewing. While many multi-user autostereoscopic displays may well be marketed as “light-field displays,” light-field displays satisfying the above-mentioned definition will probably not be in mainstream use for a number of years. On the other hand, the significant simplification enabled by the single viewpoint design of a head-mounted (near-to-eye) display makes the use of light fields for virtual- and augmented-reality head-mounted displays a much likelier scenario in the next few years.
So, we focus on head-mounted displays in the second article of this issue, written by Professor Hong Hua of the University of Arizona, called “Recent Advances in Head-Mounted Light-Field Displays for Virtual and Augmented Reality.” It describes the state of the art in three major approaches to achieving light fields with head-mounted displays – multiple focal planes, integral imaging, and multiple optical layers. Each of these approaches shows the promise of enabling a real product. At the same time, each approach clearly has tradeoffs and that means, for the foreseeable future, successful market entry will require careful design optimization for specific target segments, such as gaming, entertainment, education, training, healthcare, logistics, and banking. Even if a general consumer product that is the equivalent of the large UHD flat-panel TV is still some ways off, we can all expect to see light-field systems in different spheres of our lives in the next few years.
Goodbye, Flat World. Welcome to the Light Field! •